# 15.4 Install and configure Kafka Connect and the Adobe Experience Platform Sink Connector ## 15.4.1 Download the Adobe Experience Platform Sink Connector Go to and download the latest official release of the Adobe Experience Platform Sink Connector. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-6fc9b6e25547c525e3ae92a1cdcdd435b4434560%2Fkc1.png?alt=media) Place the download file, **streaming-connect-sink-0.0.14-java-11.jar**, onto your desktop. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-b70bdbfe57d3d5f9be68e5e6e2ac8c096f661e49%2Fkc2.png?alt=media) ## 15.4.2 Configure Kafka Connect Go to the folder on your desktop named **Kafka\_AEP** and navigate to the folder `kafka_2.13-3.1.0/config`. In that folder, open the file **connect-distributed.properties** using any Text Editor. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-d22dc50b6d16790cbdffbf069f72c499c280a8e1%2Fkc3a.png?alt=media) In your Text Editor, go to line 34 and 35 and make sure to set the fields `key.converter.schemas.enable` and `value.converter.schemas.enable` to `false` ```json key.converter.schemas.enable=false value.converter.schemas.enable=false ``` Save your changes to this file. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-061ac015d8e3d22b1238a57e460091e5ac1692aa%2Fkc3.png?alt=media) Next, go back to the folder `kafka_2.13-3.1.0` and manually create a new folder, and name it `connectors`. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-f41df1a467de404c1fd7aa487841e6b51a0023f0%2Fkc4.png?alt=media) Right-click the folder and click **New terminal at Folder**. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-4d4dfc6cd356573d6018462e8fde6e626d07af11%2Fkc5.png?alt=media) You'll then see this. Enter the command `pwd` to retrieve the full path for that folder. Select the full path and copy it to your clipboard. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-baaf617ae3ce4f78c77c7cd63138a21c3e2304c0%2Fkc6.png?alt=media) Go back to your Text Editor, to the file **connect-distributed.properties** and scroll down to the last line (line 86 in the screenshot). You should uncomment the line that starts with `# plugin.path=` and you should paste the full path to the folder named `connectors`. The result should look similar to this: `plugin.path=/Users/woutervangeluwe/Desktop/Kafka_AEP/kafka_2.13-3.1.0/connectors` Save your changes to the file **connect-distributed.properties** and close your Text Editor. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-b400e0bb1f4fc7ac6e233b7d802665cf84de54ed%2Fkc7.png?alt=media) Next, copy the latest official release of the Adobe Experience Platform Sink Connector that you downloaded into the folder named `connectors`. The file that you downloaded before is named **streaming-connect-sink-0.0.14-java-11.jar**, you can simply move it into the `connectors` folder. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-7c362aa34e0134f1f26791d0c32188ef1d8deae6%2Fkc8.png?alt=media) Next, open a new Terminal window at the level of the **kafka\_2.13-3.1.0** folder. Right-click that folder and click **New Terminal at Folder**. In the Terminal window, paste this command: `bin/connect-distributed.sh config/connect-distributed.properties` and click **Enter**. This command will start Kafka Connect and will load the library of the Adobe Experience Platform Sink Connector. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a2f61269ec3235e92536b032eb78b1419381792c%2Fkc9.png?alt=media) After a couple of seconds, you'll see something like this: ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-0d0d6d8aba61c20a48109f05b861bb2122b48a54%2Fkc10.png?alt=media) ## 15.4.3 Create your Adobe Experience Platform Sink Connector using Postman You can now interact with Kafka Connect using Postman. To do so, download [this Postman Collection](https://github.com/AdobePartnerTechnicalServices/aep-quickstart-guide/blob/main/assets/postman/postman_kafka.zip) and uncompress it to your local computer on the desktop. You'll then have a file that is called `Kafka_AEP.postman_collection.json`. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-46306a2c4112343dd54f908a36b9d2e68618ac47%2Fkc11a.png?alt=media) You need to import this file in Postman. To do so, open Postman, click **Import**, drag and drop the file `Kafka_AEP.postman_collection.json` into the popup and click **Import**. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-c6340e5aab180199e40467b328995fbe2b9e52a6%2Fkc11b.png?alt=media) You'll then find this collection in the left menu of Postman. Click the first request, **GET Available Kafka Connect connectors** to open it. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-86f49e577403994d5139eaeff17995dcb895b1af%2Fkc11c.png?alt=media) You'll then see this. Click the blue **Send** button, after which you should see an empty response `[]`. The empty response is due to the fact that no Kafka Connect connectors are currently defined. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-c07319349090513a04ed64641c72fffd5d068320%2Fkc11.png?alt=media) To create a connector, click to open the second request in the Kafka collection, **POST Create AEP Sink Connector**. You'll then see this. On line 11, where it says **"aep.endpoint": ""**, you need to paste in the HTTP API Streaming endpoint URL that you received at the end of exercise [15.3](https://docs.adobesandbox.com/comprehensive-technical-tutorial-archive/module15/ex3). The HTTP API Streaming endpoint URL looks like this: `https://dcs.adobedc.net/collection/d282bbfc8a540321341576275a8d052e9dc4ea80625dd9a5fe5b02397cfd80dc`. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-039da7c7a5e2b048aa51a52d19f06ca422998280%2Fkc12a.png?alt=media) After pasting it, the body of your request should look like this. Click the blue **Send** button to create your connector. You'll get an immediate response of the creation of your connector. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5964d014453bb978e205760291b0b18aaa7fbaf6%2Fkc12.png?alt=media) Click the first request, **GET Available Kafka Connect connectors** to open it again and click the blue **Send** button again. you'll now see that a Kafka Connect connector is created. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-8a5a48f41c04a02916578429bc4cce8168bfc5a2%2Fkc13.png?alt=media) Next, open the third request in the Kafka collection, **GET Check Kafka Connect Connector Status**. Click the blue **Send** button, you'll then get a response like the one below, stating that the connector is running. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-f797df1f164e15567061d1ef975437a2e798b57d%2Fkc14.png?alt=media) ## 15.4.4 Produce an experience event Open a new **Terminal** window by right-clicking your folder **kafka\_2.13-3.1.0** and clicking **New Terminal at Folder**. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-7aff8f7dbda6a4f675a540804c0e2884b9f97119%2Fkafka11.png?alt=media) Enter the following command: `bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic aep` ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1c84bb56e134b02b46978005f080d578db4dddfe%2Fkc15.png?alt=media) You'll then see this. Every new line followed by pushing the Enter button will result in a new message being sent into the topic **aep**. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5a1d6aabfcc4b12bdbcf968320a18366bb78274f%2Fkc16.png?alt=media) You can now send a message, which will result in being consumed by the Adobe Experience Platform Sink Connector, and which will be ingested into Adobe Experience Platform in real-time. Let's do a little demo to test this. Go to . After logging in with your Adobe ID, you'll see this. Click your website project to open it. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1f113c071919d66b9bb4e4b97d5284282e997b7a%2Fweb8.png?alt=media) On the **Screens** page, click **Run**. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-e2d1893e99187ffdd2dbf9074d64f38bb321373e%2Fweb2.png?alt=media) You'll then see your demo website open up. Select the URL and copy it to your clipboard. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9cfc19bfac7a72c64f3e750663b79c2a4acc8723%2Fweb3.png?alt=media) Open a new incognito browser window. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-48a8d19f45654fc4442dc4aad8b2a193b3f34d2f%2Fweb4.png?alt=media) Paste the URL of your demo website, which you copied in the previous step. You'll then be asked to login using your Adobe ID. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-1bb88b0513b9fac2ed6224499ac33da7d3b67d43%2Fweb5.png?alt=media) Select your account type and complete the login process. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-fbf755c12242e86c6787bb213da12d99795bd476%2Fweb6.png?alt=media) You'll then see your website loaded in an incognito browser window. For every demonstration, you'll need to use a fresh, incognito browser window to load your demo website URL. ![DSN](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-def68e96912fd147fc6d7c9c8278935f0a00fc87%2Fweb7.png?alt=media) Click the Adobe logo icon in the top left corner of your screen to open the Profile Viewer. ![Demo](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9328facfce02938dea4064997a7e05c08041ec4b%2Fpv1.png?alt=media) Have a look at the Profile Viewer panel and the Real-time Customer Profile with the **Experience Cloud ID** as the primary identifier for thi currently unknown customer. ![Demo](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a2e9cb3229702f6bfff5cabd055d614e8835a64b%2Fpv2.png?alt=media) Go to the Register/Login page. Click **CREATE AN ACCOUNT**. ![Demo](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-07768be8ef3e79b5d1256ae255909fd099b7c4f8%2Fpv9.png?alt=media) Fill out your details and click **Register** after which you'll be redirected to the previous page. ![Demo](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-bc5843e40e3bd44f91f8d36d655a629f019541fd%2Fpv10.png?alt=media) Open the Profile Viewer panel and go to Real-time Customer Profile. On the Profile Viewer panel, you should see all of your personal data displayed, like your newly added email and phone identifiers. ![Demo](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-36d01c686b8a96587318bb2337a19bb930aede27%2Fpv11.png?alt=media) You may see some experience events based on past activity. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-42c3d3f732b1e257e51954499f85ad5d5e67014f%2Fkc19.png?alt=media) Let's change that and send in a Callcenter experience event from Kafka into Adobe Experience Platform. Take the below sample experience event payload and copy it into a Text Editor. ```json { "header": { "datasetId": "61fe23fd242870194a6d779c", "imsOrgId": "--aepImsOrgID--", "source": { "name": "Launch" }, "schemaRef": { "id": "https://ns.adobe.com/experienceplatform/schemas/b0190276c6e1e1e99cf56c99f4c07a6e517bf02091dcec90", "contentType": "application/vnd.adobe.xed-full+json;version=1" } }, "body": { "xdmMeta": { "schemaRef": { "id": "https://ns.adobe.com/experienceplatform/schemas/b0190276c6e1e1e99cf56c99f4c07a6e517bf02091dcec90", "contentType": "application/vnd.adobe.xed-full+json;version=1" } }, "xdmEntity": { "eventType": "callCenterInteractionKafka", "_id": "", "timestamp": "2022-02-23T09:54:12.232Z", "_experienceplatform": { "identification": { "core": { "phoneNumber": "" } }, "interactionDetails": { "core": { "callCenterAgent": { "callID": "Support Contact - 3767767", "callTopic": "contract", "callFeeling": "negative" } } } } } } } ``` You'll then see this. You need to manually update 2 fields: * **\_id**: please set it to a random id, something like `--demoProfileLdap--1234` * **timestamp**: update the timestamp to the current date and time * **phoneNumber**: enter the phoneNumber of the account that was just created on the demo website. You can find it on the Profile Viewer panel under **Identities**. You also need to check and maybe update these fields: * **datasetId**: you need to copy the Dataset ID for the dataset Demo System - Event Dataset for Call Center (Global v1.1) * **imsOrgID**: your IMS Org ID is `--aepImsOrgId--` {% hint style="info" %} The field **\_id** needs to be unique for every data ingestion. If you produce multiple events, please make sure that you update the field **\_id** every time to a new, unique value. {% endhint %} ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-cb1b023bb535b9887bf75102027d8ceec56f0d47%2Fkc20.png?alt=media) You should then have something like this: ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-9fbd66a1ddfdbdbca59acc47a643c405e45dadb1%2Fkc21.png?alt=media) Next, copy your full experience event to your clipboard. The whitespace of your JSON payload needs to be stripped and we'll use an online tool to do that. Go to to do that. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-936172f32ab1daa5a812e2561f79ad942df6e38f%2Fkc22.png?alt=media) Paste your experience event into the editor and click **Remove white space**. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-0874e52463e00a34a4808eeba892f99de4bffd22%2Fkc22a.png?alt=media) Next, select all of the output text and copy it to your clipboard. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-7a8eed15ebcb745946de46056f0ad557b7bd6962%2Fkc23.png?alt=media) Go back to your Terminal window. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5a1d6aabfcc4b12bdbcf968320a18366bb78274f%2Fkc16.png?alt=media) Paste the new payload without whitespaces into the Terminal window and click **Enter**. ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-15e7a02f269ba81446dcc4ce221b7b95f98975ef%2Fkc23a.png?alt=media) Next, go back to your demo website and refresh the page. You should now see an experience event on your profile, under **Other Events**, just like the one below: ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-5d3a92b3e4389f8b61d29180f4f40bc46eac5703%2Fkc24.png?alt=media) {% hint style="info" %} If you want your call center interactions to appear on the Profile Viewer panel, you need to add the below label and filter in your project on , by going to the tab **Profile Viewer**. {% endhint %} ![Kafka](https://858372621-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FpBC8bA57il8Sj47B7QPJ%2Fuploads%2Fgit-blob-a4c84e004e983ef9cb2d81a0e232362061a40e0d%2Fkc25.png?alt=media) You have finished this exercise.