What you need to do
Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC) strongly encourages organisations to immediately apply available patches, available from CVE-2019-19781 - Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance.
For versions which do not currently have a patch available, ASD's ACSC strongly encourages affected organisations to immediately follow the mitigation steps provided by Citrix, available from Mitigation Steps for CVE-2019-19781.
Affected versions include:
- Citrix ADC and Citrix Gateway version 13.0 all supported builds
- Citrix ADC and NetScaler Gateway version 12.1 all supported builds
- Citrix ADC and NetScaler Gateway version 12.0 all supported builds
- Citrix ADC and NetScaler Gateway version 11.1 all supported builds
- Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds.
Detecting compromise
Check Citrix server “httpaccess.log” and “httperror.log” file for indicators of exploitation. Noting that tradecraft of this vulnerability is evolving, ASD's ACSC currently recommends looking for the following:
- POST or GET requests to paths containing “/vpns/” indicating access to potentially vulnerable resources such as “newbm.pl” and “rmbm.pl”
- GET requests which contain code such as
GET /vpns/portal/<malcious_code> - POST or GET request to XML files which have been recently created or have unusual filenames.
Note that these logs may be compressed due to file size limits or aging policies. Ensure archived versions are also checked.
If packet captures are available, a common HTTP header field used during the exploitation process is “NSC_USER”. The ASD's ACSC recommends looking for suspicious field values such as the example below:
NSC_USER: ../../../../netscaler/portal/templates/<malcious_code>
If there is any evidence of malicious activity present within the above logs, further analysis should be undertaken using the following artefacts:
- Process Listing - Look for any suspicious child processes of “httpd” owned by user “nobody.”
- File System – Look for any recently created or unusual XML files, specifically in locations which have permission to write and execute files such as:
- /netscaler/portal/templates
- /var/tmp/netscaler/portal/templates
- bash.log - Look for any suspicious executables such as curl, hostname, uname or whoami, or commands run by user “nobody”. This file contains information on command executions even if the environment variable HISTFILE has been unset.
- Scheduled Tasks – Look for cron jobs that have been created to run as user “nobody”. By default, there should be no scheduled cron jobs run as user “nobody.”
Detecting post-compromise actions
The ASD's ACSC’s analysis to date identified instances where actors have installed web shells in additional locations. This is suspected to be used as a secondary access method. The file paths observed by the ASD's ACSC include:
- var/vpn/themes/admin.php
- var/vpn/themes/default/default.php
Other filenames and directories are possible.
This web shell is a variation of the commonly used China Chopper web shell. More information on this family of web shells is available at Breaking Down the China Chopper Web Shell - Part I.
The ASD's ACSC recommends that agencies that have identified successful exploitation of their Citrix NetScaler devices analyse publicly accessible web roots for web shells. Since Citrix NetScaler devices support a variety of programming languages, the ASD's ACSC recommends all PHP, Python and Perl scripts are inspected for evidence of web shells.
Remediating compromise
If you detect compromise, we recommend that you take the following actions to remediate:
- Implement patch if available or follow the mitigations described in the Citrix support article CVE-2019-19781 - Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance
- Copy identified malicious XML files to external device and remove the original malicious files from the following directories:
- /netscaler/portal/templates
- /var/tmp/netscaler/portal/templates
- Validate all cron jobs created to run as user “nobody”. By default, there should be no scheduled cron jobs run as user “nobody.”
- Clear or reset authenticated session cookies.
- Reboot your Citrix server to disconnect any active connections from malicious actors.
- Reset passwords for all local accounts on the Citrix server.
- Perform analysis of XML files and other forensic artefacts to identify further mitigation actions.
Indicators of Compromise (IoCs) identified by the ASD's ACSC
The following list includes locations of tools that were installed post compromise:
- var/vpn/themes/admin.php
- var/vpn/themes/default/default.php
The observed instances of the above webshell each use a unique 16-character password. As such, hashes cannot be provided as a reliable indicator to assist organisations in identification efforts.
Further information
Read the Citrix Security Bulletin: CVE-2019-19781 - Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance.
Read ASD's ACSC’s guidance on how organisations can prepare for and respond to a cyber security incident.
To report a cybercrime, visit ReportCyber.
To learn more about the OAIC Notifiable Data Breaches scheme, visit the OAIC website.