serrooT/artemis-android-dynamic-traces

ARTEMIS Android Dynamic Traces

ARTEMIS executes Android applications in controlled emulators and collects dynamic analysis artifacts. This public dataset contains the runs performed in Android 10 (API 29) and Android 14 (API 34). APK binaries are not included. The public dataset is `serrooT/artemis-android-dynamic-traces`.

The dataset preserves the analysis artifacts as collected. Trace contents are not redacted, sanitized, normalized, or recompressed. Existing Zstandard files retain their original compressed bytes, plain files remain plain, unknown regular files are kept, and zero-byte files remain real zero-byte TAR members.

Repository layout

.
├── README.md
├── LICENSE
├── CITATION.md
├── release_manifest.json
├── data/
│   ├── android10/
│   │   ├── catalog.jsonl
│   │   ├── analyses.parquet
│   │   ├── artifacts.parquet
│   │   └── shards/
│   │       ├── part-00000.tar
│   │       └── ...
│   └── android14/
│       ├── catalog.jsonl
│       ├── analyses.parquet
│       ├── artifacts.parquet
│       └── shards/
│           ├── part-00000.tar
│           └── ...
└── metadata/
    ├── paper_metrics/
    │   ├── table_03_android10_storage.parquet
    │   └── table_10_published_measurements.parquet
    └── virustotal/
        └── first_seen_index.parquet

There is no year partition. Historical source directory names containing 2024 do not describe the APK population: VirusTotal first_seen values span multiple years. Future collections can add immutable shards under the appropriate Android environment in new dataset revisions; existing run identities and shards are not replaced.

android10 and android14 are execution environments, not machine-learning train/test splits.

Analysis layout inside each TAR

Every TAR preserves the natural ARTEMIS APK/run hierarchy:

<sha256>/
└── analysis_runs/
    └── <run_id>/
        ├── analysis.log
        ├── metadata.json
        └── results/                         # present when result files were collected
            ├── strace.txt[.zst]
            ├── device_logs/
            │   ├── logcat_<timestamp>.txt[.zst]
            │   ├── processes_<timestamp>.txt
            │   ├── memory_<timestamp>.txt
            │   ├── properties_<timestamp>.txt
            │   └── final_log_<timestamp>.txt/
            │       └── device.log[.zst]
            └── error_logs/
                └── device.log[.zst]

The stable public run identity is:

(environment, sha256, source_run_id)

Here, environment is android10 or android14, and source_run_id is the UUID used as <run_id> in the TAR path. The current release has one canonical run per APK in each environment where it appears. The identity supports future additional runs; runs from different environments or with different UUIDs are distinct observations.

The timestamped final_log_<timestamp>.txt path is a directory in the original collector output despite its .txt suffix. Paths and file bytes are preserved rather than renamed inside the release.

Exact run artifacts in this release

No tcpdump.pcap member is present in the current public payload. A tracer can be listed in configuration or runtime metadata even when it produced no archived file. Do not infer that every run contains a syscall trace: failed runs commonly contain only analysis.log, metadata.json, and an error device log.

Files ending in .zst contain the original Zstandard bytes and can be read without modifying the archive, for example with zstdcat file.zst. A plain counterpart is not silently preferred over a compressed one: when both existed, one was omitted only after logical equality was established; invalid or unequal pairs were retained.

metadata.json

metadata.json is the authoritative per-run description. Its principal fields are:

Because failed analyses may stop before every stage, consumers must tolerate missing optional nested fields. The JSON is preserved as collected; the release does not rewrite historical Android/API strings inside it.

Files in each Android directory

catalog.jsonl is the original line-oriented ARTEMIS result catalog, with one entry per cataloged run. Top-level fields are sha256, result, timestamp, duration, apk_path, worker_slot, and status. result records the observed outcome, error details, and source output paths. Values can contain historical absolute paths and legacy directory labels such as 2024; these strings are provenance preserved from collection time and are not claims about APK age.

analyses.parquet is a compact, typed table with one row per cataloged run. It is intended for selection and joins without opening the raw TARs. Its columns are context_id, dataset_name, sha256, source_run_id, android_version, api_level, observer, input_method, status, duration_sec, error_type, package_name, abis, and abi_parse_warning.

artifacts.parquet has one row per archived file and includes:

• —identity: Android/API, APK SHA-256, run ID, catalog status, and artifact type;
• —location: TAR path, member path, header/data byte offsets, and padded/member sizes;
• —integrity: archived-member SHA-256 and, for plain files, logical size/SHA-256;
• —source provenance: original relative path, compression, size, SHA-256, and empty-file flag; and
• —paired-file handling: peer path/checksum, pair resolution, and Zstandard validation state.

shards/*.tar contains complete runs. A run is never split across TAR files. TAR is used only to avoid publishing hundreds of thousands of small repository files. It has no outer compression because most large traces are already Zstandard-compressed and because uncompressed TAR supports efficient HTTP Range access.

Every released row currently has archive_member_present=true, a non-null member_sha256, and a location in exactly one shard. Consumers should still use the column rather than assume this contract for future release versions.

Current release scope

`release_manifest.json` records the release-level counts, file checksums, shard ranges, and source-provenance fingerprints. Each artifacts.parquet row provides the SHA-256 and TAR location of one archived member.

Paper measurement contracts

Two small files preserve the raw-storage measurements used by the SELENE paper without requiring reviewers to download approximately 746 GB of TAR shards.

`metadata/paper_metrics/table_03_android10_storage.parquet` has one row per (sha256, source_run_id, category) in the frozen Android 10 measurement. Its categories are strace, device_logs, analysis_log, metadata, and mem_proc_props; each row records file count, stored bytes, and logical (decompressed) bytes. Its 421,413 rows cover 553,908 source files and reproduce paper Table 3.

The corrected submitted-paper total is 3,032.65 GiB logically and matches the inventory after two-decimal rounding. The only remaining display difference is the device_logs row: the paper shows 66.69 GiB stored and 743.21 GiB logically, while the frozen byte-level inventory recomputes 66.68 GiB and 743.22 GiB, respectively. The frozen metric snapshot covers 111,497 run directories, whereas the final release catalog contains 111,495 canonical runs after duplicate reconciliation. It is therefore a paper-measurement contract, not a replacement for the current artifacts.parquet inventory.

`metadata/paper_metrics/table_10_published_measurements.parquet` contains ten frozen measurements: five categories for each environment over the 30,746 paired APKs. It reproduces the Android 14/Android 10 ratios in paper Table 10. Because the raw trees were organized after the paper snapshot, this file is a frozen paper contract rather than a new scan of the current directory state.

All current Android 14 APK SHA-256 values also occur in Android 10. Most Android 10 APKs do not have an Android 14 run.

The Android 14 source tree also contains 102 runs absent from the release catalog. They are excluded from the public payload and retained only in a private audit inventory. Operational CURRENT_RUN markers are not analysis artifacts and are also excluded.

Downloading one environment

The examples require Python 3.12 or newer and:

pip install huggingface_hub pandas pyarrow

The example downloads Android 14 (about 208 GB). Change both android14 occurrences to android10 for Android 10 (about 537 GB). Omit allow_patterns to download the complete dataset (about 746 GB).

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="serrooT/artemis-android-dynamic-traces",
    repo_type="dataset",
    local_dir="artemis-android14",
    allow_patterns=[
        "README.md",
        "LICENSE",
        "CITATION.md",
        "release_manifest.json",
        "data/android14/**",
    ],
)

Extracting one APK run

import tarfile

import pandas as pd
from huggingface_hub import hf_hub_download

repo_id = "serrooT/artemis-android-dynamic-traces"
environment = "android14"
sha256 = "<APK_SHA256>"
source_run_id = "<RUN_UUID>"

index_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="dataset",
    filename=f"data/{environment}/artifacts.parquet",
)
index = pd.read_parquet(index_path)
selected = index[
    index["sha256"].eq(sha256)
    & index["source_run_id"].eq(source_run_id)
]

if selected.empty:
    raise KeyError("run not found")

archive_name = selected["archive_file"].unique().tolist()
if len(archive_name) != 1:
    raise ValueError("a run must belong to exactly one TAR shard")

archive_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="dataset",
    filename=archive_name[0],
)
members = set(selected["member_path"])
with tarfile.open(archive_path, "r:") as archive:
    for member in archive:
        if member.name in members:
            archive.extract(member, path="selected-run", filter="data")

The artifact index also contains byte offsets, allowing applications to retrieve individual members with HTTP Range requests without downloading the complete shard.

Relationship to SELENE

This dataset is SELENE's raw provenance layer. The `serrooT/selene-android-paper-artifacts` dataset contains the derived paper artifacts. The public SELENE repository documents single-trace and ARTEMIS-shard processing, paper-result reproduction, and the bounded reviewer audit.

Sample age metadata

`metadata/virustotal/first_seen_index.parquet` has one row for each of the 111,495 unique APK SHA-256 values in this release. Its columns are sha256, first_seen, first_seen_year, and scan_date. All Android 14 SHA-256 values are already in the Android 10 set, so the index is not duplicated by environment.

A missing first_seen remains null; scan_date is never substituted. first_seen describes when VirusTotal first observed a sample, not when the APK was built and not when ARTEMIS executed it.

Complete VirusTotal reports are not redistributed in this repository. They contain third-party threat-intelligence results and are subject to the provider's terms. A researcher with appropriate VirusTotal access can use a SHA-256 from the index with the official API or open the current report at:

https://www.virustotal.com/gui/file/<sha256>/detection

The VirusTotal-derived columns are third-party metadata and are not relicensed by the ARTEMIS license.

License and citation

Use is permitted under the ARTEMIS Dynamic Traces Data License 1.0. Public outputs using the traces and ARTEMIS-produced metadata or indexes must cite the ARTEMIS paper as specified in CITATION.md:

Cláudio Torres Júnior, Dario Fernandes Filho, João Pincovscy, and André Grégio. “ARTEMIS: Uma Plataforma Modular para Execução, Monitoramento e Investigação de Aplicativos Android Suspeitos.” Anais do XXV Simpósio Brasileiro de Cibersegurança (SBSeg 2025), pages 147–162, 2025. https://doi.org/10.5753/sbseg.2025.11393

Responsible use

Raw traces may contain malware-controlled endpoints, identifiers, arguments, and paths from the emulated environment. They are retained as scientific observations. This dataset is intended for defensive security research; users remain responsible for applicable laws and institutional policies.

Maintainer

Cláudio Torres Júnior — Departamento de Informática, Universidade Federal do Paraná (UFPR); SecRET, Curitiba, Paraná, Brazil. Email: claudio.torres@ufpr.br.