Retrieve events

The 10Duke Enterprise system generates event data that can be used, for example, for business intelligence or audit purposes.

The event data is stored in Amazon Web Services (AWS), and you can retrieve it in JSON format, for example, to import to your reporting tool. By default, the events are stored in AWS for 90 days.

You retrieve events using the AWS Command Line Interface (AWS CLI) tool, which you need to first install and configure on your local computer.

Best practices

We recommend that you run the AWS CLI tool at regular intervals (for example, on an hourly or daily basis) and only retrieve the new events generated since the last tool run.

You can schedule the AWS CLI to run automatically using the task scheduling tools provided by your operating system.

It’s good to be aware that events generated by 10Duke Enterprise components become available in AWS after a short delay (typically a couple of minutes).

Before you start

Contact the 10Duke Integration Support team to get the following:

  • For the AWS CLI configurations:

    • AWS Identity and Access Management (IAM) user credentials configured for your 10Duke Enterprise deployment (an access key ID and a secret access key).

    • The region to be used (such as us-east-1).

  • For running the AWS CLI, the file path to the event storage in your deployment’s AWS S3 bucket.

Set up AWS CLI

Install the AWS CLI tool on your local computer, and configure your AWS IAM user credentials and other settings in the tool.

See guidelines below for a setup on a Windows or macOS computer.

AWS also provides getting started instructions for the AWS CLI in their documentation.

Set up AWS CLI on Windows

  1. First install the AWS CLI tool if you don’t have it yet. Otherwise go straight to the step for configuring the AWS CLI.

    1. On the AWS website, download the AWS CLI installer for Windows to your local computer.

    2. Run the installer by double-clicking the downloaded .msi file.

    3. Follow the on-screen instructions to complete the installation.

  2. Open the command prompt on your computer: press the Windows key + R , enter cmd and press Enter.

  3. To verify that the installation was successful, run this command to display the AWS CLI version information:

    aws --version

  4. Next, run this command to configure the AWS CLI:

    aws configure

  5. When prompted, enter the following:

    • The AWS access key ID and secret access key (your AWS IAM user credentials)

    • The default region name

    You’re also prompted for the default output format for AWS API messages: note that this setting doesn’t affect 10Duke Enterprise events, which are always in JSON format.

Now you have the AWS CLI installed and configured, and you can run it to retrieve events.

Set up AWS CLI on macOS

  1. Open the Terminal on your computer.

  2. First install the AWS CLI tool if you don’t have it yet. Otherwise go straight to the step for configuring the AWS CLI.

    Install the AWS CLI using one of the following options:

    • Homebrew (recommended):

      1. First install Homebrew if you don’t have it yet. Run this command in the Terminal:

        /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

      2. Install the AWS CLI by running this command:

        brew install awscli

    • Python’s package manager pip:

      1. First install pip if you don’t have it yet. Run this command in the Terminal:

        sudo easy_install pip

      2. Install the AWS CLI by running this command:

        pip install awscli

    • AWS CLI installer:

      1. On the AWS website, download the AWS CLI installer for macOS to your local computer.

      2. Run the installer by double-clicking the downloaded .pkg file.

      3. Follow the on-screen instructions to complete the installation.

  3. To verify that the installation was successful, run this command in the Terminal to display the AWS CLI version information:

    aws --version

  4. Next, run this command to configure the AWS CLI:

    aws configure

  5. When prompted, enter the following:

    • The AWS access key ID and secret access key (your AWS IAM user credentials)

    • The default region name

    You’re also prompted for the default output format for AWS API messages: note that this setting doesn’t affect 10Duke Enterprise events, which are always in JSON format.

Now you have the AWS CLI installed and configured, and you can run it to retrieve events.

Retrieve events using AWS CLI

Use the AWS CLI tool to copy events from your deployment’s AWS S3 bucket to a specified folder on your local computer.

In the S3 bucket, events are stored in the events-log/events/ folder, organized into a nested folder structure year/month/date/hour/ based on the event time. For example, the folder events-log/events/2024/08/26/02/ would contain data from August 26th, 2024, between 2-3 AM.

Retrieve events on Windows

  1. Open the command prompt on your computer: press the Windows key + R , enter cmd and press Enter.

  2. To copy events from the S3 bucket to your local computer, run this command:

    aws s3 cp s3://<bucket-source-path> <local-destination-path> --recursive

    • In <bucket-source-path>, provide the path to the source folder in the S3 bucket from which you want to copy data.

    • In <local-destination-path>, provide the path to the destination folder on your local computer where to store the copied data.

    • Include the --recursive option to also copy data in any subfolders in the specified source folder.

    An example command that retrieves the events from a specific hour:

    aws s3 cp s3://mybucket/events-log/events/2024/08/26/02/ C:\Users\YourUsername\Documents\local-folder --recursive

Retrieve events on macOS

  1. Open the Terminal on your computer.

  2. To copy events from the S3 bucket to your local computer, run this command:

    aws s3 cp s3://<bucket-source-path> <local-destination-path> --recursive

    • In <bucket-source-path>, provide the path to the source folder in the S3 bucket from which you want to copy data.

    • In <local-destination-path>, provide the path to the destination folder on your local computer where to store the copied data.

    • Include the --recursive option to also copy data in any subfolders in the specified source folder.

    An example command that retrieves the events from a specific hour:

    aws s3 cp s3://mybucket/events-log/events/2024/08/26/02/ /Users/yourusername/Documents/local-folder --recursive

Output format

The events copied from the AWS S3 bucket to your destination folder are in JSON format and compressed into GNU zip files (.gz).

A .gz file contains one text file with concatenated JSON event objects in the format {event1}{event2}{event3} (see example). The number of events stored in a single file varies depending on your system operations and configurations.

The folder structure from the S3 bucket is also copied. For example, if you copy events from a certain day, your destination folder will show folders per hour, each folder containing the .gz files from that hour. If you only copy data from a given hour, this gives you only .gz files.

Use the decompression tools available in your operating system to extract the JSON event files from the .gz files.

See more

On this page