Orchestration of GCE Linux’s with AWX & Ansible to Detect Keylogger Activities on Linux Endpoints with Wazuh

This article demonstrates a sample orchestration using AWX & Ansible with Linux servers hosted on GCP. The aim is to show how AWX enhances the use of Ansible in orchestrating servers across your infrastructure.

During this article, we will develop a playbook and distribute an SCA configuration that validates configuration policies by identifying poorly configured legitimate programs on monitored Linux endpoints.

This article discusses the use of the AWX Operator on Google Kubernetes Engine. To replicate this operation in your environment, AWX must be installed.

Note that Wazuh is not necessary if you only wish to verify successful file exchange via AWX & Ansible. The focus here will be primarily on AWX & Ansible related aspects, with a brief discussion on security-related concepts.

High Level Overview

Table of Contents

Introduction

• AWX.
• Wazuh.
• Keylogger.
• How LOTL Techniques Works to Log Keystrokes ?
• Ansible Roles.
• Demo Infrastructure, Prerequisites.

Preparing Configurations

• SCA Configuration.
• Playbook & Role.

AWX Core Elements

• Interface.
• Credentials.
• Inventories, Sources.
• Templates.

Implementation

• Launch.
• Results.

Conclusion

Introduction

Ansible is a useful automation tool for orchestrating multiple servers. However, to use Ansible effectively, it’s necessary to expand its capabilities. For instance, dealing with dynamically changing hosts, managing hosts as a team, or needing central management on hundreds of machines may be challenging when relying solely on Ansible. AWX is a perfect solution to extend the usability of Ansible.

AWX

AWX is an open-source, web-based user interface and automation platform. It serves as a management interface for Ansible. Its design aims to simplify the use of Ansible by providing a centralized platform for defining, scheduling, and monitoring Ansible jobs and playbooks. AWX’s advantages include:

• A user-friendly web interface, making Ansible accessible beyond command line users.
• ⭐️ ⭐️ Centralized management of Ansible configurations, playbooks, and job templates, enhancing team collaboration and control.
• ⭐️ Role-Based Access Control (RBAC) allowing tailored user access based on their role.
• Reusable job templates for standardizing Ansible jobs.
• ⭐️ Scheduling feature for automating regular tasks.
• ⭐️ Detailed logs for reviewing Ansible runs, aiding in troubleshooting and audits.
• ⭐️ ⭐️ Integration with version control systems like Git for effective playbook management.
• Configurable notifications on job completion or failure for prompt issue responses.

Wazuh

Wazuh is an open-source Security Information and Event Management (SIEM) tool. It aims to enhance an organization’s security posture by offering log analysis, intrusion detection, vulnerability detection, and other security monitoring features. Wazuh collaborates with various components to provide a comprehensive security solution.

Keylogger

A keylogger is a type of software or hardware device that records the keystrokes typed by a user on a computer keyboard. Its primary purpose is to capture entered text, including passwords, usernames, credit card numbers, and other sensitive information. Keyloggers can be used for both legitimate and malicious purposes.

How LOTL Techniques Works to Log Keystrokes ?

PAM Abuse to Log Keystrokes:PAM, a system of files and libraries handling Linux authentication, can be exploited due to its plain text communications. Adversaries may inject code into PAM to collect credentials and disrupt SSHD by disabling modules, recording keystrokes.

Recording shell history in system log: 🔴 Threat actors could use PROMPT_COMMAND or similar shell variables to execute commands each time a shell command is run. They can also change shell configuration to log each command to the system log file, typically found in /var/log/syslog or /var/log/messages.

Ansible Roles

Roles are a method to structure your playbooks in a more modular and reusable way. They allow complex tasks to be broken down into smaller, well-defined units for reuse in multiple playbooks. This promotes best practices, code reusability, and maintainability in your Ansible projects.

Since AWX is a tool built on Ansible, the repository for playbooks should be as modular as possible.

Roles have a predefined directory structure. A typical role directory structure might look like this:

role_name/
├── defaults/
├── files/
├── handlers/
├── meta/
├── tasks/
├── templates/
├── tests/
└── vars/

You can include a role using the roles keyword in a playbook.

---
- name: Example playbook with roles
  hosts: servers
  roles:
    - role_name

During the development of playbook stage you will get the point more clear.

Demo Infrastructure Requirements

To simulate AWX related parts

• AWX Operator running on GKE
• VM’s to be used as target hosts.

To simulate Wazuh related parts

• Wazuh OVA 4.7.0 that includes the Wazuh central components (Wazuh server, Wazuh indexer, and Wazuh dashboard).
• Any Linux endpoint with Wazuh agent 4.7.0 installed and enrolled to the Wazuh server.

SCA Config

We create the SCA policy on the monitored endpoint to detect when adversaries capture keystrokes using LOTL techniques.

Steps (Manual)

1- Create a new policy file /var/ossec/etc/sca_detect_linux_keylogger.yml

This configuration file defines a set of security checks aimed at identifying potential keyloggers on Linux endpoints. It leverages rules and conditions to analyze system files, configurations, and running processes.

policy:
  id: "detect_linux_keylogger"
  file: "sca_detect_linux_keylogger.yml"
  name: "System audit to detect keylogger on Linux endpoints"
  description: "Scans to detect potential keyloggers on Linux endpoints."
  references:
    - https://attack.mitre.org/techniques/T1056/001/
requirements:
  title: "Check that the SSH service and password-related files are present on the system"
  description: "Requirements for running the SCA scan against the Unix based systems policy."
  condition: any
  rules:
    - "f:$sshd_file"
    - "f:/etc/passwd"
    - "f:/etc/shadow"
variables:
  $sshd_file: /etc/ssh/sshd_config
  $pam_d_files: /etc/pam.d/common-password /etc/pam.d/passwd-auth /etc/pam.d/system-auth /etc/pam.d/system-auth-ac /etc/pam.d/passwd /etc/pam.d/sshd
  $bash_cfg_files: /root/.bashrc /root/.bash_profile /root/.bash_aliases /home/*/.bashrc /home/*/.bash_profile /home/*/.bash_aliases /etc/skel/.bashrc
  $zsh_cfg_files: /root/.zshrc /root/.zprofile /root/.zsh_aliases /home/*/.zshrc,/home/*/.zprofile /home/*/.zsh_aliases /etc/zsh/zshrc
  $tcsh_cfg_files: /root/.tcshrc /root/.login /root/.cshrc /home/*/.tcshrc /home/*/.login /home/*/.cshrc /etc/csh.cshrc
  $fish_cfg_files: /root/.config/fish/config.fish /root/.config/fish/functions/*.fish /home/*/.config/fish/config.fish /home/*/.config/fish/functions/*.fish /etc/fish/config.fish
  $ksh_cfg_files: /root/.profile /root/.kshrc /home/*/.profile /home/*/.kshrc /etc/profile /etc/kshrc
checks:
  - id: 10030
    title: "PAM abuse to log keystrokes"
    description: "Threat actors may enable the PAM auditing feature to log keystrokes."
    rationale: "By adding the line 'pam_tty_audit.so enable=' to the PAM configuration files, the system is configured to require the pam_tty_audit.so module to log TTY sessions."
    remediation: "Edit the /etc/pam.d/password-auth and /etc/pam.d/system-auth files, to disable the PAM TTY auditing feature."
    compliance:
      - pci_dss: ["2.2.4"]
      - nist_800_53: ["CM.1"]
    condition: none
    rules:
      - 'c:cat $pam_d_files 2>/dev/null -> r:^session \.*pam_tty_audit.so \.*enable='
  - id: 10031
    title: "PAM abuse to log passwords"
    description: "Threat actors may enable the PAM auditing feature to log password."
    rationale: "By a line similar to 'pam_tty_audit.so enable=* log_password' to the PAM configuration files, the system is configured to require the pam_tty_audit.so module to log passwords in TTY sessions."
    remediation: "Edit the /etc/pam.d/password-auth and /etc/pam.d/system-auth files, to disable the PAM TTY password auditing feature."
    compliance:
      - pci_dss: ["2.2.4"]
      - nist_800_53: ["CM.1"]
    condition: none
    rules:
      - 'c:cat $pam_d_files 2>/dev/null -> r:^session \.*pam_tty_audit.so \.*enable=\.*log_password'
  - id: 10032
    title: "Logging shell history to file"
    description: "Threat actors may log shell history to a file they can easily read from, most commonly a log file."
    rationale: "Threat actors utilize shell prompts such as the PROMPT_COMMAND variable, the precmd() function or some other shell feature to log every command executed."
    remediation: "Edit your shell configuration files, and remove configuration lines that attempt to write shell history to another file different from the default shell history file."
    compliance:
      - pci_dss: ["2.2.4"]
      - nist_800_53: ["CM.1"]
    condition: none
    rules:
      - 'c:cat $bash_cfg_files -> r:^PROMPT_COMMAND=\.*history -a'
      - 'c:cat $zsh_cfg_files -> r:^precmd\.*\(\)\.*history -a'
      - 'c:cat $tcsh_cfg_files -> r:^alias precmd\.*history -S'
      - 'c:cat $fish_cfg_files -> r:^function fish_prompt\.*history -h'
      - 'c:cat $ksh_cfg_files -> r:^PS1=\.*history -a'
  - id: 10033
    title: "Known keylogger process"
    description: "Threat actors may utilize known keylogging applications."
    rationale: "Threat actors may use off-the-shelf keylogging applications to log user keystrokes."
    remediation: "Investigate the endpoint, kill the running process, remove the persistence of the keylogger."
    compliance:
      - pci_dss: ["2.2.4"]
      - nist_800_53: ["CM.1"]
    condition: none
    rules:
      - 'c:ps -e -o comm= -> r:^logkeys|^lkl|^uberkey|^thc-vlogger'
      - 'c:ps aux -> r:py.*keylogger'

• SCA ID 10030 checks for keystroke logging via PAM auditing on Linux.
• SCA ID 10031 checks for password logging via PAM auditing on Linux.
• SCA ID 10032 checks if shell history is logged in the system file on Linux.
• SCA ID 10033 checks for usage of known keylogging applications on Linux.

2- Change the owner and group of the sca_detect_linux_keylogger.yml file to wazuh.

chown wazuh:wazuh /var/ossec/etc/sca_detect_linux_keylogger.yml

3- Restart the Wazuh agent to apply the changes:

systemctl restart wazuh-agent

These are steps that you can proceed manually, but making these configurations on more than 100 machines will make you unhappy :) .

Playbook & Role

The role we are preparing will utilize a single file share and common Linux commands, making this hierarchy sufficient.

modifying-service.yml/
├── modify-service/files/sca_detect_linux_keylogger.yml
├── modify-service/tasks/main.yml

modifying-service.yml

- name: Change Config and Restart Service
  hosts: all
  become: true
  gather_facts: no
  roles:
    - modify-service

• Name: “Change Config and Restart Service” — Describes the playbook.
• Hosts: all - Tasks apply to all inventory hosts.
• Become: true - Tasks run with elevated privileges, like using sudo.
• Gather Facts: no - Disables automatic fact gathering to speed up execution.
• Roles: Specifies roles to execute, referencing a role named modify-service for task organization and reuse.

main.yml

This file is pretty straightforward and understandable.

- name: Copy file to /var/ossec/etc/
  copy:
    src: sca_detect_linux_keylogger.yml
    dest: /var/ossec/etc/

- name: Change ownership of the file
  command: chown wazuh:wazuh /var/ossec/etc/sca_detect_linux_keylogger.yml

- name: Restart service wazug-agent, in all cases
  ansible.builtin.service:
    name: wazuh-agent
    state: restarted

sca_detect_linux_keylogger.yml

We already prepare this configuration, we only need to put under modify-service/files/.

This should be the final configuration;

GitHub - asrinandirin/wazuh-keylogger-config-ansible

AWX Core Elements

AWX has a very simple interface, which may seem complicated at first glance. To make everything more clear, I will guide you through the interface in the necessary sequence to demonstrate the example study.

Interface

This is a classic dashboard that gives you insight into your overall environment when you first log in to the AWX. The left panel contains the core elements that we will talk about shortly.

Inventories

When we run an ansible job, the first thing we need to do is to determine the target hosts where the job (Playbook) will be applied. In order to pull hosts dynamically, we need to be authenticated to the GCP environment.

Authentication to GCP

AWX authenticates to the GCP environment using the service account we define in the credentials section. It then automatically syncs the hosts based on the filters we specify. We will then manually create public-private key pairs and add publict key to the project metadata settings, private key to the new credentials with changing type of the key to the “machine” to connect to the VMs via SSH.

Creating Inventory

This step is pretty straightforward, no customization required. Organization section should be created when you first install AWX. It is basically for RBAC settings and teams. No custom hard settings needed for Organization.

After the inventory is created, it is necessary to configure the source as Google Compute Engine to pull the hosts that will be contained in this inventory.

Host Source Configuration

This part is very important, because we organize which hosts will be dynamically pulled. The AWX alias of the GCP Service account should be given in the Credentials section.

In the example case, I want all hosts in 3 GCP projects that I labeled as “linux” to be pulled, so you need to filter the Source Variables section as follows below.

Source variables

---
projects:
  - project-ID-1
  - project-ID-2
  - project-ID-3

hostnames:
  # List host by name instead of the default public ip
  # - name -> When you typed that VM's pulled by Hostnames, It sounds nice but 
            # It could be a problem if your GKE can't solve hostnames via dns. 
            # so I suggest not removing the comment line..

# I assume your VM's has linux labels. You can specify as your organizations 
# labeling standarts. 
filters:
  - labels.platform = linux 

After that, save your changes and sync your hosts. In the hosts tab you can find your hosts, you can turn them on or off according to your needs.

Now that the inventory is ready, we need to connect repository with AWX, to do that we need to create project.

Do not forget, each new revision needs a new sync between project and repository manually.

Manual sync.

Now the last step before the launch is creation a job template to connect project and inventory.

Template

Implementation

Launch

So far, we have created all the necessary elements. Now, we are ready to execute the job. You can also set up notifications for various jobs and create templates for recurring tasks.

Supported Notification Channels

We have one error because one of the vm’s is off but everything else is correctly worked. Let’s check it manually.

Results

To view the results immediately in Wazuh Dashboard, restart the Wazuh agent to trigger the SCA scan which will detect the keylogger IoCs:

Below is an example image of SCA results in the Security configuration assessment tab of the Wazuh dashboard when Wazuh detects keylogging activities in a Linux endpoint.

Conclusion

In this article, I demonstrated a simple automation that can be performed on Google Compute Engine VMs with Ansible & AWX to secure the environment with Wazuh.

Ansible has a really wide use; it can provide you with flexibility in many areas from application deployments to infrastructure management. (Unlike Terraform, Ansible does not provide clear state management, so I do not recommend using it for infrastructure management.) However, in many cases where state is not required, your GCP Resources can be managed very well with the GCP modules in Ansible Galaxy.