minkyuchoi/Temporal-Logic-Video-Dataset

Temporal Logic Video (TLV) Dataset

<!-- PROJECT LOGO --> <br /> <div align="center"> <h3 align="center">Temporal Logic Video (TLV) Dataset</h3> <p align="center"> Synthetic and real video dataset with temporal logic annotation <br /> <a href="https://github.com/UTAustin-SwarmLab/temporal-logic-video-dataset"><strong>Explore the GitHub »</strong></a> <br /> <br /> <a href="https://anoymousu1.github.io/nsvs-anonymous.github.io/">NSVS-TL Project Webpage</a> · <a href="https://github.com/UTAustin-SwarmLab/Neuro-Symbolic-Video-Search-Temploral-Logic">NSVS-TL Source Code</a> </p> </div>

Overview

The Temporal Logic Video (TLV) Dataset addresses the scarcity of state-of-the-art video datasets for long-horizon, temporally extended activity and object detection. It comprises two main components:

1. 1.Synthetic datasets: Generated by concatenating static images from established computer vision datasets (COCO and ImageNet), allowing for the introduction of a wide range of Temporal Logic (TL) specifications.
2. 2.Real-world datasets: Based on open-source autonomous vehicle (AV) driving datasets, specifically NuScenes and Waymo.

Table of Contents

• —Dataset Composition
• —Dataset
• —License

Dataset Composition

Synthetic Datasets

• —Source: COCO and ImageNet
• —Purpose: Introduce artificial Temporal Logic specifications
• —Generation Method: Image stitching from static datasets

Real-world Datasets

• —Sources: NuScenes and Waymo
• —Purpose: Provide real-world autonomous vehicle scenarios
• —Annotation: Temporal Logic specifications added to existing data

Dataset

Though we provide a source code to generate datasets from different data sources, we release a dataset v1 as a proof of concept.

Dataset Structure

We provide a v1 dataset as a proof of concept. The data is offered as serialized objects, each containing a set of frames with annotations.

File Naming Convention

\<tlv_data_type\>:source:\<datasource\>-number_of_frames:\<number_of_frames\>-\<uuid\>.pkl

Object Attributes

Each serialized object contains the following attributes:

• —ground_truth: Boolean indicating whether the dataset contains ground truth labels
• —ltl_formula: Temporal logic formula applied to the dataset
• —proposition: A set of propositions for ltl_formula
• —number_of_frame: Total number of frames in the dataset
• —frames_of_interest: Frames of interest which satisfy the ltl_formula
• —labels_of_frames: Labels for each frame
• —images_of_frames: Image data for each frame

You can download a dataset from here. The structure of the dataset is as follows: serializer.

   tlv-dataset-v1/
   ├── tlv_real_dataset/
   ├──── prop1Uprop2/
   ├──── (prop1&prop2)Uprop3/
   ├── tlv_synthetic_dataset/
   ├──── Fprop1/
   ├──── Gprop1/
   ├──── prop1&prop2/
   ├──── prop1Uprop2/
   └──── (prop1&prop2)Uprop3/

Dataset Statistics

1. 1.Total Number of Frames

1. 1.Total Number of datasets

License

This project is licensed under the MIT License. See the LICENSE file for details.

Connect with Me

<p align="center"> <em>Feel free to connect with me through these professional channels:</em> </p> <div style="display: flex; justify-content: center; align-items: center; flex-wrap: nowrap;"> <a href="https://www.linkedin.com/in/mchoi07/" target="blank"><img src="https://img.shields.io/badge/LinkedIn-0077B5?style=flat-square&logo=linkedin&logoColor=white" alt="LinkedIn" style="margin: 0 5px;"/></a> <a href="mailto:minkyu.choi@utexas.edu"><img src="https://img.shields.io/badge/Email-D14836?style=flat-square&logo=gmail&logoColor=white" alt="Email" style="margin: 0 5px;"/></a> <a href="https://scholar.google.com/citations?user=ai4daB8AAAAJ&hl" target="blank"><img src="https://img.shields.io/badge/Scholar-4285F4?style=flat-square&logo=google-scholar&logoColor=white" alt="Google Scholar" style="margin: 0 5px;"/></a> <a href="https://minkyuchoi-07.github.io" target="blank"><img src="https://img.shields.io/badge/Website-00C7B7?style=flat-square&logo=internet-explorer&logoColor=white" alt="Website" style="margin: 0 5px;"/></a> <a href="https://x.com/MinkyuChoi7" target="blank"><img src="https://img.shields.io/badge/Twitter-1DA1F2?style=flat-square&logo=twitter&logoColor=white" alt="Twitter" style="margin: 0 5px;"/></a> </div>

Citation

If you find this repo useful, please cite our paper:

@inproceedings{Choi_2024_ECCV,
  author={Choi, Minkyu and Goel, Harsh and Omama, Mohammad and Yang, Yunhao and Shah, Sahil and Chinchali, Sandeep},
  title={Towards Neuro-Symbolic Video Understanding},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  month={September},
  year={2024}
}