bbmb/deep-learning-for-embedding-model-ssilwal-qpham6_army_doc

deep learning project 2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on the json dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: sentence-transformers/all-MiniLM-L6-v2 <!-- at revision fa97f6e7cb1a59073dff9e6b13e2715cf7475ac9 -->
• —Maximum Sequence Length: 256 tokens
• —Output Dimensionality: 384 dimensions
• —Similarity Function: Cosine Similarity
• —Training Dataset:
• —json
• —Language: en
• —License: apache-2.0

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bbmb/deep-learning-for-embedding-model-ssilwal-qpham6_army_doc")
# Run inference
sentences = [
    'Offense \n11 January 2024 ATP 3-21.8 4-61\nlight the target, making it easier to acquire effectively. Leaders and Soldiers \nuse the infrared devices to identify enemy or friendly personnel and then \nengage targets using their aiming lights. \n4-172. Illuminating rounds fired to burn on the ground can mark objectives. This helps\nthe platoon orient on the objective but may adversely affect night vision devices.\n4-173. Leaders plan but do not always use illumination during limited visibility\nattacks. Battalion commanders normally control conventional illumination but ma y\na\nuthorize the company commander to do so. If the commander decides to use\nconventional illumination , the commander should not call for it until the assault is\ninitiated or the attack is detected. It should be placed on several locations over a wide\narea to confuse the enemy as to the exact place of the attack. It should be placed beyond\nthe objective to help assaulting Soldiers see and fire at withdrawing or counterattacking\nenemy Soldiers. Infrared illumination is a good capability to light the objective without\nlighting it for enemy forces without night vision devices.  This advantage is degraded\nwhen used against a peer threat with the same night vision capabilities.\n4-174. The platoon leader , squad leaders , and vehicle commanders must know unit\ntactical SOP and develop sound COAs to synchronize the employment of infrared\nillumination devices , target designators , and aiming lights during their assault on the\nobjective. These include using luminous tape or chemical lights to mark personnel and\nusing weapons control restrictions.\n4-175. The platoon leader may use the following techniques to increase control during\nthe assault:\n\uf06c Use no flares, grenades, or obscuration on the objective.\n\uf06c Use mortar or artillery rounds to orient attacking units.\n\uf06c Use a base squad or fire team to pace and guide others.\n\uf06c Reduce intervals between Soldiers and squads.\n4-176. Like a daylight attack , indirect and direct fires are planned for a limited\nvisibility attack but are not executed unless the platoon is detected or is ready to assault.\nSome weapons may fire before the attack and maintain a pattern to deceive the enemy\nor to help cover noise ma de by the platoon ’s movement. This is not done if it will\ndisclose the attack.\n4-177. Obscuration further reduces the enemy’s visibility, particularly if the enemy has\nnight vision devices. The FO fires obscuration rounds close to or on enemy positions ,\nso it does not restrict friendly movement or hinder the reduction of obstacles. Employing \nobscuration on the objective during the assault may make it hard for assaulting Soldiers\nto find enemy fighting positions. If enough thermal sights are available , obscuration on\nthe objective may provide a decisive advantage for a well-trained platoon.\nNote. I f the enemy is equipped with night vision devices , leaders must evaluate \nthe risk of using each technique and ensure the mission is not compromised by \nthe enemy’s ability to detect infrared light sources.',
    'What are the advantages of using infrared illumination in assaults?',
    'How can leaders effectively provide command and control during defensive operations?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Information Retrieval

• —Datasets: dim_384, dim_256, dim_128 and dim_64
• —Evaluated with <code>InformationRetrievalEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

json

• —Dataset: json
• —Size: 4,820 training samples
• —Columns: <code>positive</code> and <code>anchor</code>
• —Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:--------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details | <ul><li>min: 100 tokens</li><li>mean: 248.18 tokens</li><li>max: 256 tokens</li></ul> | <ul><li>min: 9 tokens</li><li>mean: 15.06 tokens</li><li>max: 27 tokens</li></ul> |
• —Samples: | positive | anchor | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------| | <code>Appendix A <br>A-22 ATP 3-21.8 11 January 2024 <br>A-68. Observed fire. Usually is used when the platoon is in protected defensive positions <br>with engagement ranges more than 2,500 meters for stabilized systems (when attached)<br>and 1,500 meters for unstabilized systems. It can be employed between elements of the<br>platoon, such as the squad lasing and observing while the weapons squad engages. The<br>platoon leader directs one squad to engage. The remaining squads observe fires and<br>prepare to engage on order in case the engaging element consistently misses its targets ,<br>experiences a malfunction, or runs low on ammunition. Observed fire allows for mutual<br>observation and assistance while protecting the location of the observing elements.<br>A-69. Sequential fire. Entails the subordinate elements of a unit engaging the same point <br>or area target one after another in an arranged sequence. Sequential fire also can help to<br>prevent the waste of ammunition, as when a platoon waits to see the effects of the ...</code> | <code>What is the purpose of having one squad engage while others observe in an observed fire scenario?</code> | | <code>Glossary <br>Glossary-4 ATP 3-21.8 11 January 2024 <br>PLD probable line of deployment <br>PPEP personal protective equipment posture <br>RFL restrictive fire line <br>RM risk management <br>ROE rules of engagement <br>RS reduced sensitivity <br>RTO radiotelephone operator <br>S-2 battalion or brigade intelligence staff officer <br>SALUTE size, activity, location, unit, time, and equipment <br>SDM squad-designated marksman <br>SITEMP situation template <br>SLM shoulder-launched munition <br>SOP standard operating procedure <br>STP Soldier training publication <br>TAA tactical assembly area <br>TC training circular <br>TCCC tactical combat casualty care <br>TLP troop leading procedures <br>TM technical manual <br>TRP target reference point <br>U.S. United States <br>WARNORD warning order <br>WCS weapons control status <br>WP white phosphorous <br>SECTION II – TERMS <br>actions on contact <br>A process to help leaders understand what is happening and to take action. <br>(FM 3-90) <br>air-ground operations <br>The simultaneous or synchronized employment of ground forces with avi...</code> | <code>How is the term SDM used in the military?</code> | | <code>Chapter 1 <br>1-2 ATP 3-21.8 11 January 2024 <br>MISSION, CAPABILITIES, AND LIMITATIONS <br>1-2. The mission of the Infantry rifle platoon is to close with the enemy using fire and<br>movement to destroy or capture enemy forces , or to repel enemy attacks by fire , close<br>co<br>mbat, and counterattack to control land areas , including populations and resources.<br>The Infantry rifle platoon leader exercises command and control and directs the<br>operation of the platoon and attached units while conducting combined arms warfare<br>throughout the depth of the platoon’s area of operations (AO). Platoon missions ,<br>although not inclusive, may include reducing fortified areas , infiltrating and seizing<br>objectives in the enemy’ s rear, eliminating enemy force remnants in restricted terrain ,<br>securing key facilities and activities, and conducting operations in support of stability<br>operations tasks in the wake of maneuvering forces. Reconnaissance and surveillance<br>operations and security operations remain a core compe...</code> | <code>What offensive and defensive actions can an Infantry rifle platoon perform?</code> |
• —Loss: <code>MatryoshkaLoss</code> with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: epoch
• —per_device_train_batch_size: 64
• —per_device_eval_batch_size: 16
• —gradient_accumulation_steps: 8
• —num_train_epochs: 20
• —lr_scheduler_type: cosine
• —warmup_ratio: 0.2
• —bf16: True
• —tf32: True
• —load_best_model_at_end: True
• —optim: adamwtorchfused
• —batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: epoch
• —prediction_loss_only: True
• —per_device_train_batch_size: 64
• —per_device_eval_batch_size: 16
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 8
• —eval_accumulation_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.0
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 20
• —max_steps: -1
• —lr_scheduler_type: cosine
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.2
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: True
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: True
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: True
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamwtorchfused
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: False
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —prompts: None
• —batch_sampler: no_duplicates
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

• —The bold row denotes the saved checkpoint.

Framework Versions

• —Python: 3.10.12
• —Sentence Transformers: 3.3.1
• —Transformers: 4.41.2
• —PyTorch: 2.1.2+cu121
• —Accelerate: 0.34.2
• —Datasets: 2.19.1
• —Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->