---
analysis: A simple rule might catch a Python shell, but this hunt pivots from specific
  staging paths to rare persistence mechanisms and uses stack-counting to differentiate
  malicious systemd services from legitimate fleet-wide management, providing context
  a single rule cannot.
blind_spots:
- id: limited-process-telemetry
  question: What specifically was submitted via the Ray Jobs API?
  requires: Deep inspection of Ray job submission logs
  risk: Attackers can submit Python code that executes purely in-memory without spawning
    observable shell child processes.
  stage: initial-access-ray-job-agent
- id: limited-file-telemetry
  question: Was the kernel rootkit successfully compiled and loaded?
  requires: hb_linux_audit for DKMS build monitoring
  risk: Once the rootkit is loaded, it can manipulate process and file listings, making
    the presence of the toolkit invisible to standard user-space queries.
  stage: persistence-toolkit-deployment
coverage:
- stage: initial-access-ray-job-agent
  status: covered
  steps:
  - python-reverse-shells
- stage: execution-reverse-shell
  status: covered
  steps:
  - python-reverse-shells
- stage: persistence-toolkit-deployment
  status: covered
  steps:
  - staging-path-lead
  - rare-systemd-persistence
- reason: 'Belongs to another part of the ''0% GPU utilization: How SANSAR hides Linux
    cryptomining'' series.'
  stage: defense-evasion-stealth-mechanisms
  status: out_of_scope
- reason: 'Belongs to another part of the ''0% GPU utilization: How SANSAR hides Linux
    cryptomining'' series.'
  stage: c2-tor-and-ethereum
  status: out_of_scope
- reason: 'Belongs to another part of the ''0% GPU utilization: How SANSAR hides Linux
    cryptomining'' series.'
  stage: impact-cryptomining
  status: out_of_scope
guardrails:
  claims: no_unsupported
  evidence: citation_required
  missing_data: not_benign
  telemetry: untrusted
hunt:
  applicability: campaign-specific
  handoff: promote-to-detection
  justification: Exploitation of AI/ML orchestration tools like Ray is an active vector
    for cryptomining. Detecting the staging and initial reverse shell is the only
    way to catch this intrusion before the kernel rootkit is loaded and evades standard
    EDR monitoring.
  methodology: model-assisted
  trigger: intel-report
hypothesis: An attacker has exploited an unauthenticated Ray Job API to execute a
  Python reverse shell and stage the SANSAR toolkit under /opt/vadi_/, including persistence
  via SSH and systemd.
labels:
- hunt
- attack.t1059.004
- attack.t1059.006
- attack.t1543.002
- attack.t1098.004
- command and control
- defense evasion
- execution
- impact
- initial access
- persistence
name: Ray Framework Exploitation and SANSAR Staging
parameters:
  lookback_days:
    default: '14'
    description: Days of history to examine.
    from:
      kind: manual
      observed: '2024-11-12'
      ref: default-retention
    type: number
  scope_hosts:
    default: []
    description: Target hosts identified in the lead query; leave empty to scan the
      entire estate.
    from:
      kind: manual
      observed: '2024-11-12'
      ref: analyst-scoping
    type: list[host]
  staging_path:
    default: /opt/vadi_/
    description: The reported staging path for the SANSAR toolkit.
    from:
      kind: article
      observed: '2024-11-12'
      ref: REF3364
    type: path
provenance:
  authors:
  - name: Huntbase hunt generation
    org: huntbase.io
  generated:
    by: huntbase-hunt-generation
    from: https://www.elastic.co/security-labs/threat-command/sansar-linux-cryptomining-malware
    gates:
    - dry-run
    - lint
    - critic
    model: hb_google/gemini-3-flash-preview
rationale: Start by identifying hosts running Ray (look for processes named 'ray'
  or 'gcs_server'). The lead query focuses on the unique staging path /opt/vadi_/
  to quickly isolate affected nodes before performing heavier behavioral checks.
references:
- name: "Elastic Security Labs \u2014 0% GPU utilization: How SANSAR hides Linux cryptomining"
  url: https://www.elastic.co/security-labs/threat-command/sansar-linux-cryptomining-malware
related:
- hunt: sansar-c2-and-evasion
  reason: This hunt focuses on the initial access; the next hunt in the series focuses
    on the Tor-based C2 and rootkit stealth mechanisms.
  relation: follows
scenario:
  stages:
  - name: Ray Job Agent Exploitation
    observables:
    - Unauthenticated access to Ray Job API
    - External probes in Ray logs
    - Job submission to Ray head node
    slug: initial-access-ray-job-agent
    tactic: initial-access
    techniques:
    - T1059
  - name: Python Reverse Shell Execution
    observables:
    - nvidia-smi --query-gpu=name
    - Python reverse-shell job submission
    - Connection to external listener via Python
    slug: execution-reverse-shell
    tactic: execution
    techniques:
    - T1059
  - name: Persistence and Toolkit Deployment
    observables:
    - Staging under /opt/vadi_/
    - Modification of authorized_keys for SSH access
    - netns backdoor registered as a systemd service
    - acpi_thermal_mgr kernel module built via DKMS
    - initramfs loader modification
    slug: persistence-toolkit-deployment
    tactic: persistence
    techniques:
    - T1059
  - name: Stealth and Telemetry Falsification
    observables:
    - acpi_thermal_mgr.ko loading
    - Replacement of libnvidia-ml.so.1 with modified wrapper
    - Presence of libnvidia-ml.so.1.backup
    - SANSAR anti-debugging TracerPid check in /proc/self/status
    slug: defense-evasion-stealth-mechanisms
    tactic: defense-evasion
    techniques:
    - T1059
  - name: Evasive Command and Control
    observables:
    - efxisnqldelyr5ra25oqfrsut54b34g26e7zrsiuupsoekjof2gho2id.onion:9051
    - api.ipify.org
    - ifconfig.me
    - icanhazip.com
    - checkip.amazonaws.com
    - Lyrebird transport helper usage (obfs4, WebTunnel)
    - Ethereum smart contract calls for netns C2 resolution
    slug: c2-tor-and-ethereum
    tactic: command-and-control
    techniques:
    - T1090.003
  - name: Cryptomining Resource Hijacking
    observables:
    - kurt (XMRig 6.25.0) process activity
    - tilki (SRBMiner-MULTI 3.6.9) process activity
    - Deployment of GPU mining contexts
    slug: impact-cryptomining
    tactic: impact
    techniques:
    - T1496
  summary: Threat actor REF3364 targets Linux GPU hosts by exploiting unauthenticated
    Ray job agents to gain root access and persistent shell control. The campaign
    deploys the SANSAR toolkit, which employs a kernel rootkit to hide its presence
    and modified NVIDIA libraries to falsify GPU utilization while managing cryptomining
    and backdoors via Tor and Ethereum-based infrastructure.
series:
  index: 1
  slug: 0-gpu-utilization-how-sansar-hides-linux-cryptomining
  title: '0% GPU utilization: How SANSAR hides Linux cryptomining'
  total: 2
severity: high
targets:
  analyst:
    name: Tier-2 analyst
    role: analyst
  endpoint:
    category: endpoint
    name: Endpoint telemetry (hb_ surfaces)
    telemetry:
    - endpoint
  hunter:
    agent: true
    name: Hunt agent
tlp: clear
type: investigation
---


# Ray Framework Exploitation and SANSAR Staging

This hunt targets the initial access and persistence phases of the REF3364 SANSAR intrusion. It searches for indicators of Ray Job Agent abuse, specifically focusing on suspicious shell processes spawned from Python workloads and the subsequent deployment of the SANSAR toolkit in specific staging directories. The hunt uses a gated flow to first confirm the presence of the unique staging path before initiating a broader search for rare systemd persistence and reverse shell behavior across the estate.

## staging-path-lead
<!-- Lead: SANSAR Staging Directory Activity -->
Identify any process activity or file access involving the specific staging directory used by the threat actor to establish a lead.

```sqlite target=endpoint role=scoping params=(staging_path=staging_path, lookback_days=lookback_days)
~~~yaml
expected: Processes executing from or referencing the /opt/vadi_/ directory. Silence
  suggests this specific campaign staging has not occurred on the estate.
reads:
- device_hostname
- process_cmd_line
- process_name
- process_path
- time
- user_name
silence: not_evidence_of_absence
source: hb_process_activity
verified: dry-run
verified_at: '2026-11-13'
~~~
SELECT device_hostname, process_name, process_cmd_line, user_name, time FROM hb_process_activity WHERE (LOWER(process_path) LIKE '{{staging_path}}%' OR LOWER(process_cmd_line) LIKE '%{{staging_path}}%') AND time >= datetime('now', '-{{lookback_days}} days')
```

## evaluate-staging-lead
<!-- Evaluate Staging Lead -->
```agent target=hunter
cite: required
context:
- staging-path-lead
max_iterations: 3
objective: Determine if the processes found in the staging path are consistent with
  the SANSAR orchestrator or related miners.
success_criteria: A clear decision to open the gated queries or terminate the hunt.
tools:
- endpoint
```

## gate-check
<!-- Gate: Staging Found -->
if~: "The evaluate-staging-lead verdict confirms suspicious activity or unauthorized directory creation" (confidence: high, judge=hunter)
then: → deep-dive
indeterminate: → analyst-review
unavailable: → analyst-review (blind_spot: limited-process-telemetry)
else: → close-out

## deep-dive
<!-- Deep Dive into Ray Exploitation -->
parallel:
- → rare-systemd-persistence
- → python-reverse-shells
join: → final-triage

## rare-systemd-persistence
<!-- Prevalence: Rare Systemd Services -->
Identify rare systemd service files created on the estate, which may correspond to the netns backdoor.

```sqlite target=endpoint role=baseline params=(lookback_days=lookback_days, scope_hosts=scope_hosts)
~~~yaml
baseline:
  compare: first_seen
  window: '{{lookback_days}}d'
expected: New systemd unit files appearing on only a few hosts. Legit services are
  usually fleet-wide.
prevalence:
  by: device_hostname
  key:
  - file_path
  rare_below: 3
reads:
- activity_id
- device_hostname
- file_path
- time
silence: not_evidence_of_absence
source: hb_file_activity
verified: dry-run
verified_at: '2026-11-13'
~~~
SELECT file_path, COUNT(DISTINCT device_hostname) AS hosts, MIN(time) AS first_seen FROM hb_file_activity WHERE (file_path LIKE '/etc/systemd/system/%' OR file_path LIKE '/usr/lib/systemd/system/%') AND activity_id = 1 AND ('{{scope_hosts}}' = '' OR instr(',' || '{{scope_hosts}}' || ',', ',' || device_hostname || ',') > 0) AND time >= datetime('now', '-{{lookback_days}} days') GROUP BY file_path HAVING hosts <= 3
```

## python-reverse-shells
<!-- Behaviour: Python Reverse Shells from Ray -->
Identify Python processes exhibiting reverse shell syntax that are spawned from a Ray or web service context.

```sqlite target=endpoint role=detection-candidate params=(lookback_days=lookback_days, scope_hosts=scope_hosts)
~~~yaml
expected: Python shells spawned by non-interactive accounts. This confirms the initial
  access phase of the intrusion.
reads:
- device_hostname
- parent_process_name
- process_cmd_line
- process_name
- time
- user_name
silence: not_evidence_of_absence
source: hb_process_activity
verified: dry-run
verified_at: '2026-11-13'
~~~
SELECT device_hostname, process_name, process_cmd_line, parent_process_name, user_name, time FROM hb_process_activity WHERE (LOWER(process_cmd_line) LIKE '%python%' AND (LOWER(process_cmd_line) LIKE '%socket%' OR LOWER(process_cmd_line) LIKE '%pty.spawn%')) AND ('{{scope_hosts}}' = '' OR instr(',' || '{{scope_hosts}}' || ',', ',' || device_hostname || ',') > 0) AND time >= datetime('now', '-{{lookback_days}} days')
```

## final-triage
<!-- Final Triage and Correlation -->
```agent target=hunter
cite: required
context:
- evaluate-staging-lead
- rare-systemd-persistence
- python-reverse-shells
max_iterations: 6
objective: Determine if the host has been compromised via Ray and whether the SANSAR
  toolkit is actively persisting and executing shell commands.
success_criteria: A verdict of malicious | suspicious | benign per host, citing relevant
  rows.
tools:
- endpoint
```

## route-verdict
<!-- Route Final Verdict -->
if~: "the final-triage verdict is malicious for at least one host" (confidence: high, judge=hunter)
then: → isolate-host
indeterminate: → analyst-review
unavailable: → analyst-review (blind_spot: limited-file-telemetry)
else: → close-out

## isolate-host
<!-- Isolate Compromised Host -->
```action target=endpoint
~~~yaml
approval: required
~~~
Isolate the host, terminate the identified Python reverse shell processes, and preserve the staging directory for forensic analysis.
```
→ analyst-review

## analyst-review
<!-- Analyst Review -->
```manual target=analyst
Verify the presence of the /opt/vadi_/ directory and any new SSH keys in authorized_keys. Confirm if the Ray head node dashboard or job agent is reachable from the internet.
```
→ end

## close-out
<!-- Hunt Close-out -->
```manual target=analyst
Record whether any compromised hosts were found and note any Ray frameworks that require hardening.
```
→ end