# 2.4 Data Ingestion from Offline Sources
In this exercise, the goal is to onboard external data like CRM Data in Platform.
## Learning Objectives
* Learn how to generate test data
* Learn how to ingest CSV
* Learn how to use the web UI for data ingestion through Workflows
* Understand the data governance features of Experience Platform
## Resources
* Mockaroo UI:
* Experience Platform UI:
## Tasks
* Create a CSV file with demo date. Ingest the CSV file in Adobe Experience Platform by making use of the available workflows.
* Understand data governance options in Adobe Experience Platform
## 2.4.1 Create your CRM Dataset through a data generator tool
For this you need 1000 sample lines of CRM Data.
Open the Mockaroo Template by going to .
On the template, you'll notice the following fields:
* id
* first\_name
* last\_name
* email
* gender
* birthDate
* home\_latitude
* home\_longitude
* country\_code
* city
* country
All these fields have been defined to produce data that is compatible with Platform.
To generate your CSV-file, click the **Download Data** button which will give you a CSV-file with 1000 lines of demo-data.
Open your CSV-file in Microsoft Excel to visualize its contents.
With your CSV-file ready, you can proceed with mapping it against XDM.
### 2.4.2 Verify the CRM Onboarding Dataset in Adobe Experience Platform
Open [Adobe Experience Platform](https://experience.adobe.com/platform) and go to **Datasets**.
Before you continue, you need to select a **sandbox**. The sandbox to select is named `--module2sandbox--`. You can do this by clicking the text **Production Prod** in the blue line on top of your screen. After selecting the appropriate sandbox, you'll see the screen change and now you're in your dedicated sandbox.
In Adobe Experience Platform, click on **Datasets** in the menu on the left side of your screen.
You're going to use a shared dataset based in this enablement. The shared dataset has been created already and is called **Demo System - Profile Dataset for CRM (Global v1.1)**.
Open the dataset **Demo System - Profile Dataset for CRM (Global v1.1)**.
On the overview screen, you can see 3 main pieces of information.
First of all, the Dataset Activity dashboard shows the total number of CRM records in the dataset and the ingested batches and their status
Second, by scrolling down on the page you can check when batches of data were ingested, how many records were onboarded and also, whether or not the batch was successfully onboarded. The **Batch ID** is the identifier for a specific batch job, and the **Batch ID** is important as it can be used for troubleshooting why a specific batch was not successfully onboarded.
Lastly, the Dataset Info tab shows important information like the Dataset ID (again, important from a troubleshooting perspective), the Dataset's Name and whether the dataset was enabled for Profile.
The most important setting here is the link between the dataset and the Schema. The Schema defines what data can be ingested and how that data should look like.
In this case, we're using the **Demo System - Profile Schema for CRM (Global v1.1)**, which is mapped against the class of **Profile** and has implemented extensions, also called field groups.
By clicking on the name of the schema, you're taken to the Schema overview were you can see all the fields that have been activated for this schema.
Every schema needs to have a custom, primary descriptor defined. In the case of our CRM dataset, the schema has defined that the field **crmId** should be the primary identifier. If you want to create a schema and link it to the Real-time Customer Profile, you need to define a custom Field Group that refers to your primary descriptor.
In the above screenshot, you can see that our descriptor is located in `--aepTenantId--.identification.core.crmId`, which is set as the Primary Identifier, linked to the namespace of **Demo System - CRMID**.
Every schema and as such, every dataset that should be used in the Real-time Customer Profile should have one Primary identifier. This Primary identifier is the identifier user by the brand for a customer in that dataset. In the case of a CRM dataset it might be the email-address or the CRM ID, in the case of a Call Center dataset it might be the mobile number of a customer.
It is best practice to create a separate, specific schema for every dataset and to set the descriptor for every dataset specifically to match how the current solutions used by the brand operate.
### 2.4.3 Using a workflow to map a CSV file to an XDM Schema
The goal of this is to onboard CRM data in Platform. All the data that is ingested in Platform should be mapped against the specific XDM Schema. What you currently have is a CSV dataset with 1000 lines on the one side, and a dataset that is linked to a schema on the other side. To load that CSV file in that dataset, a mapping needs to take place. To facilitate this mapping exercise, we have **Workflows** available in Adobe Experience Platform.
The workflow that we'll use here, is the workflow named **Map CSV to XDM Schema** in the Data Ingestion menu.
Click the **Map CSV to XDM Schema** button. Click **Launch** to start the process.
On the next screen, you need to select a dataset to ingest your file in. You have the choice between selecting an already existing dataset or creating a new one. For this exercise, we'll reuse an existing one: please select **Demo System - Profile Dataset for CRM (Global v1.1)** as indicated below and leave the other settings set to default.
Click **Next** to go to the next step.
Drag & Drop your CSV-file or click **Browse** and navigate on your computer to your desktop and select your CSV-file.
After selecting your CSV-file it will upload immediately and you will see a preview of your file within seconds.
Click **Next** to go to the next step. It can take a few seconds while the file is processed completely.
You now need to map your CSV Column Headers with an XDM-property in your **Demo System - Profile Dataset for CRM**.
Adobe Experience Platform has already made some proposals for you, by trying to link the Source Attributes with the Target Schema Fields.
For the Schema Mappings, Adobe Experience Platform has tried to link fields together already. However, not all proposals of mapping are correct. You now need to **Accept target fields** one-by-one.
#### birthDate
The Source Schema field **birthDate** should be linked to the target field **person.birthDate**.
#### city
The Source Schema field **city** should be linked to the target field **homeAddress.city**.
#### country
The Source Schema field **country** should be linked to the target field **homeAddress.country**.
#### country\_code
The Source Schema field **country\_code** should be linked to the target field **homeAddress.countryCode**.
#### email
The Source Schema field **email** should be linked to the target field **personalEmail.address**.
#### crmid
The Source Schema field \*\* crmid\*\* should be linked to the target field **`--aepTenantId--`.identification.core.crmId**.
#### first\_name
The Source Schema field **first\_name** should be linked to the target field **person.name.firstName**.
#### gender
The Source Schema field **gender** should be linked to the target field **person.gender**.
#### home\_latitude
The Source Schema field **home\_latitude** should be linked to the target field **homeAddress.\_schema.latitude**.
#### home\_longitude
The Source Schema field **home\_longitude** should be linked to the target field **homeAddress.\_schema.longitude**.
#### id
The Source Schema field **id** should be linked to the target field **\_id**.
#### last\_name
The Source Schema field **last\_name** should be linked to the target field **person.name.lastName**.
You should now have this:
Click the **Finish** button to finish the workflow.
After clicking **Finish**, you'll then see the **Dataflow** overview, and after a couple of minutes you can refresh your screen to see if your workflow completed successfully. Click your **Target dataset name**.
You'll then see the dataset where your ingestion has processed.
On the dataset, you'll see a Batch ID that has been ingested just now, with 1000 records ingested and a status of **Success**.
Click on the **Preview Dataset**- button to get a quick view of a small sample of the dataset to ensure that the loaded data is correct.
Once data is loaded, you can define the correct data governance approach for our dataset.
### 2.5.4 Adding data governance to your dataset
Now that your customer data is ingested, you need to make sure that this dataset is properly governed for usage and export control. Click on the **Data Governance** tab and observe that you can set three types of restrictions: Contractual, Identity, and Sensitive Data.
You can find more info on the different labels and how they will be enforced in the future through the policy framework on this link:
Let's restrict identity data for the entire dataset. Hover over your dataset name, and click the Pencil icon to edit the settings.
Go to **Identity Data** and you'll see that the **I2** option is checked - this will assume that all pieces of information in this dataset are at least indirectly identifiable to the person.
Click **Save Changes** and observe that **I2** is now set for all data fields in the dataset.
You can also set these flags for individual data fields - for example, the **firstName** field is likely to be classified as an **I1** level for directly identifiable information.
Select the field **firstName** by checking the checkbox and click on **Edit Governance Labels** in the upper right corner of your screen.
Go to **Identity Data** and you'll see that the **I2** option is already checked (inherited from the dataset). The field firstName also has a field-specific configuration and is set as **I1 - Directly Identifiable Data**.
With this, you've now successfully ingested and classified CRM Data in Adobe Experience Platform.