introspector/rust-analyser

Rust-Analyzer Semantic Analysis Dataset

This dataset contains comprehensive semantic analysis data extracted from the rust-analyzer codebase using our custom rust-analyzer integration. It captures the step-by-step processing phases that rust-analyzer performs when analyzing Rust code.

Dataset Overview

This dataset provides unprecedented insight into how rust-analyzer (the most advanced Rust language server) processes its own codebase. It contains 500K+ records across multiple semantic analysis phases.

What's Included

• —Parsing Phase: Syntax tree generation, tokenization, and parse error handling
• —Name Resolution Phase: Symbol binding, scope analysis, and import resolution
• —Type Inference Phase: Type checking, inference decisions, and type error detection

Dataset Statistics

• —Total Records: ~533,000 semantic analysis events
• —Source Files: 1,307 Rust files from rust-analyzer codebase
• —Data Size: ~450MB in efficient Parquet format
• —Processing Phases: 3 major compiler phases captured

Dataset Structure

Each record contains:

• —id: Unique identifier for the analysis event
• —file_path: Source file being analyzed
• —line, column: Location in source code
• —phase: Processing phase (parsing, nameresolution, typeinference)
• —element_type: Type of code element (function, struct, variable, etc.)
• —element_name: Name of the element (if applicable)
• —syntax_data: JSON-serialized syntax tree information
• —symbol_data: JSON-serialized symbol resolution data
• —type_data: JSON-serialized type inference information
• —source_snippet: The actual source code being analyzed
• —context_before/context_after: Surrounding code context
• —processing_time_ms: Time taken for analysis
• —rust_version, analyzer_version: Tool versions used

Use Cases

Machine Learning Applications

• —Code completion models: Train on parsing and name resolution patterns
• —Type inference models: Learn from rust-analyzer's type inference decisions
• —Bug detection models: Identify patterns in diagnostic data
• —Code understanding models: Learn semantic analysis patterns

Research Applications

• —Compiler optimization: Analyze compilation patterns across large codebases
• —Language design: Study how developers use Rust language features
• —IDE improvement: Understand common semantic analysis patterns
• —Static analysis: Develop better code analysis tools

Educational Applications

• —Rust learning: Understand how code is processed step-by-step
• —Compiler education: Visualize semantic analysis phases
• —Code analysis tutorials: Interactive examples of language server internals

Data Quality

• —✅ Schema validated: All records follow consistent structure
• —✅ Data integrity: No corrupted or malformed records
• —✅ Completeness: All processed files represented
• —✅ Self-referential: rust-analyzer analyzing its own codebase

Technical Details

• —Format: Parquet files for efficient storage and fast loading
• —Compression: Snappy compression for optimal performance
• —Chunking: Files split to stay under 10MB for Git LFS compatibility
• —Schema: Strongly typed with proper null handling

Source

This dataset was generated by analyzing the rust-analyzer codebase (version 0.3.2000) using our custom integration that captures semantic analysis at multiple processing phases.

Source Project: /home/mdupont/2025/06/27/rust-analyzer Generated: August 2025 Tool: Custom rust-analyzer semantic extractor

Citation

If you use this dataset in your research, please cite:

@dataset{rust_analyzer_semantic_2025,
  title={Rust-Analyzer Semantic Analysis Dataset},
  author={Dupont, J. Mike},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/introspector/rust-analyser}
}

License

This dataset is released under the AGPL-3.0 license, consistent with the rust-analyzer project.

Acknowledgments

• —Built using the rust-analyzer project
• —Generated with custom semantic analysis extraction tools
• —Optimized for machine learning and research applications