# 14.5 Forward events towards the AWS ecosystem {% hint style="warning" %} Completion of this exercise is **optional** and a cost is involved to use AWS Kinesis. While AWS provides a free tier account which lets you test and configure many services without a cost, AWS Kinesis isn't part of that free tier account. So in order to implement and test this exercise, a cost will be involved to use AWS Kinesis. {% endhint %} ## Good to know Adobe Experience Platform supports various Amazon services as destination. Kinesis and S3 are both [profile export destinations](https://experienceleague.adobe.com/docs/experience-platform/destinations/destination-types.html?lang=en) and can be used as part of Adobe Experience Platform's Real-Time CDP. You can easily feed high-value segment events and associated profile attributes into your systems of choice. In this note, you’ll learn how to setup your own Amazon Kinesis stream to stream event data coming from the Adobe Experience Platform Edge ecosystem to a cloud storage destination, such as Amazon S3. This is useful in case you'd like to collect experience events from web and mobile properties and push them into your datalake for analysis and operational reporting. Datalakes generally ingest data in a batch fashion with large daily file imports, they do not expose public http endpoint which could be used in conjunction with event forwarding. Supporting the above use cases imply that streamed data need to be buffered or placed in a queue before being written to a file. Care has to be taken to not open file for write access across multiple process. Delegating this task to dedicated system is ideal to scale nicely while ensuring a great level of service, this is where Kinesis comes to the rescue. Amazon Kinesis Data Streams focuses on ingesting and storing data streams. Kinesis Data Firehose focuses on delivering data streams to select destinations, such as S3 buckets. As part of this exercise, you'll... * Perform a basic setup of a Kinesis data stream * Create a Firehose delivery stream and use S3 bucket as destination * Configure Amazon API gateway as a rest api endpoint to receive your event data * Forward raw event data from Adobe's Edge to your Kinesis stream ## 14.5.1 Configure your AWS S3 bucket Go to and sign in with the Amazon-account you previously created. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9f38d0008610f73401e363a56c42a87e1ef3669e%2Fawshome.png?alt=media) After logging in, you'll be redirected to the **AWS Management Console**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9203f96e43c1be48079d118c8de08cdf75c95d2a%2Fawsconsole.png?alt=media) In the **Find Services** menu, search for **s3**. Click the first search result: **S3 - Scalable Storage in the Cloud**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-12ed32c68c1e08af39d50e715ab1b90cfbb8410a%2Fawsconsoles3.png?alt=media) You'll then see the **Amazon S3** homepage. Click **Create Bucket**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5252e6c0abe6758facb58bfa798a84b3897d4f0e%2Fs3home.png?alt=media) In the **Create Bucket** screen, you need to configure two things: * Name: use the name `eventforwarding---demoProfileLdap--`. As an example, in this exercise the bucket name is **aepmodulertcdpvangeluw** * Region: use the region **EU (Frankfurt) eu-central-1** ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-c1bda65d05109652887a7e7b1f67380786eb6d43%2Fbucketname.png?alt=media) Leave all the other default settings as they are. Scroll down and click **Create bucket**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-594cbd35c78fc9f8344e5a6f71f447db8e661dcf%2Fcreatebucket.png?alt=media) You'll then see your bucket being created and will be redirected to the Amazon S3 homepage. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-beea3227a9d2f54a1e27fc1a21412b32e2ff68d3%2FS3homeb.png?alt=media) ## 14.5.2 Configure your AWS Kinesis Data Stream In the **Find Services** menu, search for **kinesis**. Click the first search result: **Kinesis - Work with Real-Time Streaming Data**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1dad1592d6e9bfc3ad6cb5998fcb12c95db93246%2Fkinesis1.png?alt=media) Select **Kinesis Data Streams**. Click **Create data stream**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a5bf31f72ad8eeb8a255ab3c8472bc29db328862%2Fkinesis2.png?alt=media) For the **Data stream name**, use `--demoProfileLdap---datastream`. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a0fd80f507b86edfa6b6522e68581c239359ef8b%2Fkinesis3.png?alt=media) There's no need to change any of the other settings. Scroll down and click **Create data stream**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1984119e5f57ea2bfb1e52f7cc84b3292cb5a4d7%2Fkinesis4.png?alt=media) You'll then see this. Once your data stream is succesfully created, you can move forward to the next exercise. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-bb5e3546ac273791730ddd8251fa86a94b295af8%2Fkinesis5.png?alt=media) ## 14.5.3 Configure your AWS Firehose Delivery Stream In the **Find Services** menu, search for **kinesis**. Click **Kinesis Data Firehose**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-3719c4f0647370312a2de0ffa17cad1e61892560%2Fkinesis6.png?alt=media) Click **Create delivery stream**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-3dca382e78c773ba97ac9e798df410ed182e90f0%2Fkinesis7.png?alt=media) For **Source**, select **Amazon Kinesis Data Streams**. For **Destination**, select **Amazon S3**. Click **Browse** to select your data stream. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-656da416a5dd83822b038b0ab10fde1ffb3f93b4%2Fkinesis8.png?alt=media) Select your data stream. Click **Choose**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-843393f15e656f29cf8df8d80e0c6a24b48f00a7%2Fkinesis9.png?alt=media) You'll then see this. Remember the **Delivery stream name** as you'll need it later. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9ce78014d91bc5a57061c853f23933d8eff464f3%2Fkinesis10.png?alt=media) Scroll down until you see **Destination Settings**. Click **Browse** to select your S3 bucket. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-e4919c2bd834aaf491c58307519ab95d55facfef%2Fkinesis11.png?alt=media) Select your S3 bucket and click **Choose**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-ee9d19838dac1a31bc0209768fcd0f9be281a205%2Fkinesis12.png?alt=media) You'll then see something like this. Update the following settings: * Dynamic partitioning: set to **Enabled** * Multi record deaggregation: set to **Disabled** * New line delimiter: set to **Enabled** * Inline parsing for JSON: set to **Enabled** ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-34d33b21c48dba9a67ad813cf334f25164a3f327%2Fkinesis13.png?alt=media) Scroll down a bit, you'll then see this. Update the following settings: * Dynamic partitionning keys * Key name: **dynamicPartitioningKey** * JQ expression: **.dynamicPartitioningKey** * S3 bucket prefix: add the following code: ```bash !{partitionKeyFromQuery:dynamicPartitioningKey}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/} ``` * S3 bucket error ouput prefix: set to **error** ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-bd531ee5fa6a5bf648ee082b5389af5b817185ef%2Fkinesis14.png?alt=media) Finally, scroll down a bit more and click **Create delivery stream** ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9c908e09e2d1052b23edc270ecc7c8ad5ccb0a99%2Fkinesis15.png?alt=media) After a couple of minutes, your delivery stream will nbe created and **Active**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1a2bbb1db6d101a62ac34f400be996193bb53a01%2Fkinesis16.png?alt=media) ## 14.5.4 Configure your AWS IAM Role In the **Find Services** menu, search for **iam**. Click **API Gateway**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fd879d07c0432e4fe1265bff22c6c1378bdcb321%2Fiam1.png?alt=media) Click **Roles**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-912e44c22e0b1b6241f2a6d951e7d061d877b2b5%2Fiam2.png?alt=media) Search for your **KinesisFirehose** role. Click it to open it. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fc7de212b7f271a72a10f87ecaaab9a210f01384%2Fiam3.png?alt=media) Click your Permissions Policy name to open it. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9ca6c885e6c767b48597194741f075b4aa55e979%2Fiam4.png?alt=media) In the new screen that opens, click **Edit Policy**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-b624fc4e7ed582ce33e5c1ad2d72adaf58273323%2Fiam5.png?alt=media) Under **Kinesis** - **Actions**, ensure that the **Write** permissions for **PutRecord** is enabled. Click **Review Policy**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-cda850cd82a707303d3d1810c04bb103c0ee0df7%2Fiam6.png?alt=media) Click **Save Changes**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-8c01ecbe33c4ec787068707c27725ca50b54f8a2%2Fiam7.png?alt=media) You'll then be back here. Click **Roles**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-540f957c8b83c7c3a430da5539f139a2150174d3%2Fiam8.png?alt=media) Search for your **KinesisFirehose** role. Click it to open it. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fc7de212b7f271a72a10f87ecaaab9a210f01384%2Fiam3.png?alt=media) Go to **Trust relationships** and click **Edit trust policy**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-7a859dc30e91c2d97fe350503bf7d7070213526d%2Fiam9.png?alt=media) Overwrite the current trust policy by pasting this code to replace the existing code: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "firehose.amazonaws.com", "kinesis.amazonaws.com", "apigateway.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] } ``` Click **Update policy** ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a6f9ef5c665bd6051f9c31b6fb0400c3899160ec%2Fiam10.png?alt=media) You'll then see this. You'll need to specify the **ARN** for this role in the next step. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a0e3a2dbc19629c06d48507082b107b597613b0d%2Fiam11.png?alt=media) ## 14.5.5 Configure your AWS API Gateway Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale. API developers can create APIs that access AWS or other web services, as well as data stored in the AWS Cloud. You will now expose the Kinesis data stream to the internet through a HTTPS endpoint which can then directly be consumed by Adobe services, like Event Forwarding. In the **Find Services** menu, search for **api gateway**. Click **API Gateway**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-66dd61b1970953d8d89bcde0fe32dda4744f3daa%2Fkinesis17.png?alt=media) You'll then see something like this. Click **Create API**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-3e70e7d749da7f78c0263288e9b989f02b8ece7a%2Fkinesis18.png?alt=media) Click **Build** on the **REST API** card. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-57a27dd99f06db6953720fc23ac96b91516a0dcb%2Fkinesis19.png?alt=media) You'll then see this. Fill out the settings like this: * Choose the protocol: select **REST** * Create new API: select **New API** * Settings: * API name: use `--demoProfileLdap---eventforwarding` * Endpoint Type: select **Regional** Click **Create API**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5a77e7d3b6095e3b8273c32da353695041506a66%2Fkinesis20.png?alt=media) You'll then see this. Click **Actions** and then click **Create Resource**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-e05df3318d820343f95a35102bcd27dfadb9b42a%2Fkinesis21.png?alt=media) You'll then see this. Set **Resource Name** to **stream**. Click **Create Resource**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-676e0d4f3293fcb59bb03aadc76e0fc58b826256%2Fkinesis22.png?alt=media) You'll then see this. Click **Actions** and then click **Create Method**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-8bed072463fb437cd18e6f3aed54e9eae1fe769a%2Fkinesis23.png?alt=media) In the dropdown, select **POST** and click the **v** button. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-f3a92e2a83ea62c04b735da7b273ca95dbec2241%2Fkinesis24.png?alt=media) You'll then see this. Fill out the settings like this: * Integration type: **AWS Service** * AWS Region: select the region that is used by your Kinesis Data Stream, in this case: **us-west-2** * AWS Service: select **Kinesis** * AWS Subdomain: leave empty * HTTP Method: select **POST** * Action Type: select **Use action name** * Action: enter **PutRecord** * Execution role: paste the **ARN** of the execution role that is used by your Kinesis Data Firehose, as instructed in the previous exercise * Content Handling: select **Passthrough** * Use Default Timeout: enable the checkbox Click **Save**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-6493f31e2cd6280746a1ce262fc2a8f06f29bb9d%2Fkinesis25.png?alt=media) You'll then see this. Click **Integration Request**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-3c74f8e6d961352b0802997aaa9f4557b4925870%2Fkinesis27.png?alt=media) Click **HTTP Headers**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-181b0dcde3f115d001e9277c1e35341ea463404b%2Fkinesis28.png?alt=media) Scroll down a bit and click **Add header**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-7d67a2227ca7a885eae9f6082f8604e15fd11f69%2Fkinesis29.png?alt=media) Set **Name** to **Content-Type**, set **Mapped from** to `'application/x-amz-json-1.1'`. Click the **v** icon to save your changes. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5affcef3db1af95e9c92cebb9cfc3bc5d3dfd435%2Fkinesis30.png?alt=media) You'll then see this. For **Request body passthrough**, select **When there are no templates defined (recommended)**. Next, click **Add mapping template**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-d6e9f635229ac82b687ccb2c8178d512ba889bc5%2Fkinesis31.png?alt=media) Under **Content-Type**, enter **application/json**. Click the **v** icon to save your changes. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-ea9d5f13bcb1235713011bf11ee4ec946b900de4%2Fkinesis32.png?alt=media) Scroll down to find a code editor window. Paste the below code in there: ```json { "StreamName": "$input.path('StreamName')", "Data": "$util.base64Encode($input.json('$.Data'))", "PartitionKey": "$input.path('$.PartitionKey')" } ``` Click **Save**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-6e0947862ec7e530e0ec0491900d4da1ab938c50%2Fkinesis33.png?alt=media) Next, scroll up and click **<- Method Execution** to go back. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-d89c45b5c37bfc51667fe7d888f2f82f62229b34%2Fkinesis34.png?alt=media) Click **TEST**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-f9302d368128f11ec477981597687bea03ab6d48%2Fkinesis35.png?alt=media) Scroll down, and paste this code under **Request Body**. Click **Test**. ```json { "Data": { "message": "Hello World", "dynamicPartitioningKey": "v2" }, "PartitionKey": "1", "StreamName": "--demoProfileLdap---datastream" } ``` ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-110ca74a736e560498a8fd6ce4f3455070422efc%2Fkinesis36.png?alt=media) You'll then see a similar result: ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-2102cc3d133caa3537b3422abe9e36c57be3d107%2Fkinesis37.png?alt=media) You'll then see this. Click **Actions** and then click **Deploy API**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-89d3fc016479fc63d07a39187143bcc3f4e655a2%2Fkinesis38.png?alt=media) For **Deployment stage**, select **New Stage**. As **Stage name**, enter **prod**. Click **Deploy**. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-35326bd3d65e3b3cf7a7f72cbb3c882abf11e107%2Fkinesis39.png?alt=media) You'll then see this. Click **Save Changes**. FYI: the URL in the image is the URL to use to send data towards (in this example: ). ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-c25feaca7858027573259e4a03b1dace07487192%2Fkinesis40.png?alt=media) You can test your setup by using the below cURL request, all you need to do is replace the below URL by yours, `https://vv1i5vwg2k.execute-api.us-west-2.amazonaws.com/prod` in this example, and add `/stream` at the end of the URL. ```json curl --location --request POST 'https://vv1i5vwg2k.execute-api.us-west-2.amazonaws.com/prod/stream' \ --header 'Content-Type: application/json' \ --data-raw '{ "Data": { "userid": "--demoProfileLdap--@adobe.com", "firstName":"--demoProfileLdap--", "offerName":"10% off on outdoor gears", "offerCode": "10OFF-SPRING", "dynamicPartitioningKey": "campaign" }, "PartitionKey": "1", "StreamName": "--demoProfileLdap---datastream" }' ``` Paste the above updated code in a Terminal window, and hit enter. You'll then see this response, similar to the response you could see when testing above. ![ETL](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-15956961ed28f7ed632c554616cfdcd1cff3771b%2Fkinesis41.png?alt=media) ## 14.5.6 Update your Event Forwarding property You can now activate to your AWS Kinesis data stream through AWS API Gateway, so you can now send your raw experience events into the AWS ecosystem. Using Real-Time CDP Connections and Event Forwarding, you can now easily enable event forwarding to your newly created AWS API Gateway endpoint. ### 14.5.6.1 Update your Event Forwarding property: Create a Data Element Go to and go to **Event Forwarding**. Search your Event Forwarding property and click it to open it. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-85f9e8fbea4cea603f8625257d720d0869df3877%2Fprop1.png?alt=media) In the left menu, go to **Data Elements**. Click **Add Data Element**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-db4d9a7484ff5bb12cd09556163b3d2667a64686%2Fdeaws1.png?alt=media) You'll then see a new data element to configure. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-53489597ebc643040233f8136f32710658730350%2Fde2.png?alt=media) Make the following selection: * As the **Name**, enter **awsDataObject**. * As the **Extension**, select **Core**. * As the **Data Element Type**, select **Custom Code**. You'll now have this. Click **\ Open Editor**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-dbf1bedace580b0ead84f5b2479db3b30287f261%2Fdeaws3.png?alt=media) In the Editor, paste the following code on line 3. Click **Save**. ```javascript const newObj = {...arc.event.xdm, dynamicPartitioningKey: "event_forwarding"} return JSON.stringify(newObj); ``` ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fc8bed9aa4ec51af5f94f0ff062f768a708c74db%2Fdeaws4.png?alt=media) {% hint style="info" %} In the above path, a reference is made to **arc**. **arc** stands for Adobe Resource Context and **arc** always stands for the highest available object that is available in the Server Side context. Enrichments and transformations may be added to that **arc** object using Adobe Experience Platform Data Collection Server functions. In the above path, a reference is made to **event**. **event** stands for a unique event and Adobe Experience Platform Data Collection Server will always evaluate every event individually. Sometimes, you may see a reference to **events** in the payload sent by Web SDK Client Side, but in Adobe Experience Platform Data Collection Event Forwarding, every event is evaluated individually. {% endhint %} You'll then be back here. Click **Save** or **Save to Library**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-862cc38471095d1c201a6a7bb9505b6fd437baa9%2Fdeaws5.png?alt=media) ### 14.5.6.2 Update your Adobe Experience Platform Data Collection Server property: Update your Rule In the left menu, go to **Rules**. Click to open the rule **All Pages** which you created in one of the previous exercises. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-8a7fb9818f43169b802a20ad4e460aad18f5a4cb%2Frlaws1.png?alt=media) You'll then see this. Click the **+** icon to add a new action. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9055e266f30e16463058404f6f884c6d6c91574d%2Frlaws2.png?alt=media) You'll then see this. Make the following selection: * Select the **Extension**: **Adobe Cloud Connector**. * Select the **Action Type**: **Make Fetch Call**. That should give you this **Name**: **Adobe Cloud Connector - Make Fetch Call**. You should now see this: ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-331f4f708c58f92fc33e09ca9aedc905c7ad5786%2Frl4.png?alt=media) Next, configure the following: * Change the request method from GET to **POST** * Enter the URL of the AWS API Gateway endpoint you created in one of the previous steps, which looks like this: `https://vv1i5vwg2k.execute-api.us-west-2.amazonaws.com/prod/stream` You should now have this. Next, go to **Headers**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-796aefaecebae3b6a18b2dac0d945b5a86c10e2c%2Frlaws5.png?alt=media) Under headers, add a new header with key **Content-Type** and value **application/json**. Next, go to **Body**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-8bba1b8113e8587b1dbe5dc6ac4298e65c9a0207%2Frlaws5a.png?alt=media) You'll then see this. Paste the following code in the field **Body (Raw)**. Click **Keep Changes**. ```json { "Data":{{awsDataObject}}, "PartitionKey": "1", "StreamName": "--demoProfileLdap---datastream" } ``` ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fe26f267080ef7bf003f8008e3d9c9af0c4ed5f6%2Frlaws7.png?alt=media) You'll then see be back here. Click **Save** or **Save to Library**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-154617ef64071024a3c39ff8a46ac35217f35ab9%2Frlaws10.png?alt=media) You've now configured your first rule in an Event Forwarding property. Go to **Publishing Flow** to publish your changes. Open your Development library by clicking **Main**. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-b5074fd0262ff1a42f6626ec8601a53e81ba7667%2Frlaws11.png?alt=media) Click the **Add All Changed Resources** button, after which you'll see your Rule and Data Element changes appear in this library. Next, click **Save & Build for Development**. Your changes are now being deployed. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5036f32134e9c391b156d62129886f9bea98fce2%2Frlaws13.png?alt=media) After a couple of minutes, you'll see that the deployment is done and ready to be tested. ![Adobe Experience Platform Data Collection SSF](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-8df687359b21ff18671f77a703bf3a527118620d%2Frl14.png?alt=media) ## 14.5.7 Test your configuration Go to . After logging in with your Adobe ID, you'll see this. Click your website project to open it. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1f113c071919d66b9bb4e4b97d5284282e997b7a%2Fweb8.png?alt=media) You can now follow the below flow to access the website. Click **Integrations**. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-90b71802742bd7e63b57d893018aa19265b77749%2Fweb1.png?alt=media) On the **Integrations** page, you need to select the Data Collection property that was created in exercise 0.1. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5cf49094f77ead0a2e2ab5581e2799517999779a%2Fweb2.png?alt=media) You'll then see your demo website open up. Select the URL and copy it to your clipboard. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9cfc19bfac7a72c64f3e750663b79c2a4acc8723%2Fweb3.png?alt=media) Open a new incognito browser window. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-48a8d19f45654fc4442dc4aad8b2a193b3f34d2f%2Fweb4.png?alt=media) Paste the URL of your demo website, which you copied in the previous step. You'll then be asked to login using your Adobe ID. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1bb88b0513b9fac2ed6224499ac33da7d3b67d43%2Fweb5.png?alt=media) Select your account type and complete the login process. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fbf755c12242e86c6787bb213da12d99795bd476%2Fweb6.png?alt=media) You'll then see your website loaded in an incognito browser window. For every demonstration, you'll need to use a fresh, incognito browser window to load your demo website URL. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-def68e96912fd147fc6d7c9c8278935f0a00fc87%2Fweb7.png?alt=media) When you open up your browser Developer View, you can inspect Network requests as indicated below. When you use the filter **interact**, you'll see the network requests that are sent by Adobe Experience Platform Data Collection Client to the Adobe Edge. ![Adobe Experience Platform Data Collection Setup](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-e62e5818da367ccb5cd94312ea3af9b613dc9129%2Fhook1.png?alt=media) If you select the raw payload, go to and paste the payload. Click **Make Pretty**. You'll then see the JSON payload, the **events** object and the **xdm** object. In one of the previous steps, when you defined the Data Element, you used the reference **arc.event.xdm**, which will result in you parsing out the **xdm** object of this payload. ![Adobe Experience Platform Data Collection Setup](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-291e0b59925ce8a072eda891f4b0fedd94ecd76e%2Fhook2.png?alt=media) Switch your view to **AWS**. By opening your data stream and going into the **Monitoring** tab, you'll now see incoming traffic. ![Adobe Experience Platform Data Collection Setup](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-46bbc3ac3ec0a4be71b2602544fdd9620082fb66%2Fhookaws3.png?alt=media) When you then open your delivery stream and go into the **Monitoring** tab, you'll also see incoming traffic. ![Adobe Experience Platform Data Collection Setup](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-b405cba160f188032a38255597fb0b165e7c3dd1%2Fhookaws4.png?alt=media) Finally, when you have a look at your S3 bucket, you'll now notice files being created there as a consequence of your data ingestion. ![Adobe Experience Platform Data Collection Setup](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-24d5c041ca79b4a54803e60b45fb31fd88a48aa9%2Fhookaws5.png?alt=media) When you download such a file and open it using a text editor, you'll see that it contains the XDM payload from the events that were forwarded. ![Adobe Experience Platform Data Collection Setup](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-c5f17868abb77e5b56c79894e711f1eabc9bbb64%2Fhookaws6.png?alt=media)