peyterho/climatebert-macro-sentiment

ClimateBERT Macro Sentiment

A fine-tuned climatebert/distilroberta-base-climate-sentiment (82M params, DistilRoBERTa) for 3-class sentiment analysis on financial, macroeconomic, and climate/ESG text.

This model is the ClimateBERT head within the larger macro-sentiment-finbert ensemble pipeline. When text is detected as climate/ESG-related, the pipeline's topic router automatically selects this model for inference. It can also be used standalone for any financial sentiment task.

Why ClimateBERT?

The base model (climatebert/distilroberta-base-climate-f) was further pre-trained on climate-related text by Bingler et al. (2023), then fine-tuned on the climatebert/climate_sentiment dataset for climate risk/neutral/opportunity classification. This gives it a strong inductive bias for climate and ESG language — terminology like stranded assets, transition risk, green bonds, and net-zero commitments are well-represented in its pre-training distribution.

By further fine-tuning on a broader mix of 5 financial sentiment datasets, we retain the climate domain advantage while gaining general financial sentiment capability. The result: the smallest model in the ensemble (82M) that still reaches 88.9% accuracy, and the best OOD generalization on financial phrasebank data among all three heads relative to its parameter count.

Quick Start

Standalone Usage

from transformers import pipeline

pipe = pipeline("text-classification", model="peyterho/climatebert-macro-sentiment", top_k=None)

# Climate/ESG text — the model's home turf
pipe("Renewable energy investments surged as solar costs fell to record lows.")
# [[{'label': 'opportunity', 'score': 0.95}, {'label': 'neutral', 'score': 0.04}, {'label': 'risk', 'score': 0.01}]]

pipe("Climate change poses significant physical risks to coastal infrastructure.")
# [[{'label': 'risk', 'score': 0.91}, {'label': 'neutral', 'score': 0.07}, {'label': 'opportunity', 'score': 0.02}]]

# General financial text — also works well after fine-tuning
pipe("Q3 earnings beat expectations with revenue up 12% year-over-year.")
# [[{'label': 'opportunity', 'score': 0.88}, ...]]

pipe("The company reported a significant decline in operating margins.")
# [[{'label': 'risk', 'score': 0.85}, ...]]

As Part of the Full Pipeline

Within the macro-sentiment-finbert pipeline, this model is automatically selected when climate/ESG keywords are detected:

from macro_sentiment import MacroSentimentPipeline

pipe = MacroSentimentPipeline(device="cpu")

result = pipe("The EU's carbon border adjustment mechanism will reshape trade flows for emission-intensive industries.")
print(result.summary())
# Sentiment: Negative (-0.215) | Policy: Neutral (+0.000) | Crisis: Normal (0.000) | Domain: climate | Climate: Risk (exp=0.45)
print(result.head_used)
# "climate"

result = pipe("Green bond issuance hit $500 billion as investors pivoted toward sustainable fixed income.")
print(result.summary())
# Sentiment: Positive (+0.534) | Policy: Neutral (+0.000) | Crisis: Normal (0.000) | Domain: climate | Climate: Opportunity (exp=0.38)

Label Mapping

This model uses ClimateBERT's native label scheme, which maps to the unified 3-class sentiment as follows:

Note: The label ordering differs from the unified schema (0=negative, 1=neutral, 2=positive). A label remapping is applied during training and inference within the ensemble pipeline. If using the model standalone, interpret opportunity as positive and risk as negative.

Evaluation Results

In-Domain (Combined Test Set — 4,333 samples)

All three fine-tuned heads compared on the same held-out test split:

Out-of-Domain: Financial Phrasebank (785 samples)

Evaluated on Jean-Baptiste/financial_news_sentiment_mixte_with_phrasebank_75 — not in the training mix. This dataset combines PhraseBank sentences (75% annotator agreement) with financial news articles.

ClimateBERT edges out FinBERT (109M) on this OOD benchmark despite having 25% fewer parameters — likely due to DistilRoBERTa's stronger pre-training on diverse text compared to BERT's financial-domain pre-training.

Out-of-Domain: Stock News Headlines (30,150 samples)

Evaluated on ic-fspml/stock_news_sentiment — 5-class labels mapped to 3-class. Not in the training mix.

Lower performance on stock news headlines is expected — these are short, noisy texts with a 5→3 class mapping. FinBERT's financial pre-training gives it an advantage on this particular domain.

Comparison Against Baselines

Training Data

Fine-tuned on 20,034 training samples combined from 5 public financial/climate sentiment datasets:

All datasets unified to 3-class: 0=negative, 1=neutral, 2=positive (then remapped to ClimateBERT's native risk/neutral/opportunity labels during training).

Training Details

Training Curve

Validation loss bottomed at epoch 3, while accuracy continued improving marginally through epoch 6. The best checkpoint (epoch 3) was loaded for the final model — the later epochs show mild overfitting on val loss despite higher accuracy, suggesting the model memorizes easy-to-classify samples while losing calibration on harder boundary cases.

Role in the Ensemble Pipeline

This model serves as the climate/ESG specialist in the macro-sentiment-finbert pipeline. The topic router activates it when climate-related keywords are detected in the input text (e.g., carbon, emission, renewable, ESG, net-zero, climate risk, green bond, Paris Agreement).

Input Text → Topic Router → climate keywords detected? → ClimateBERT (this model)
                          → policy keywords detected?  → RoBERTa-Large
                          → default / financial news    → FinBERT
                          → non-English text            → XLM-RoBERTa

The full pipeline then fuses this model's output with dictionary-based signals (Loughran-McDonald, Henry, Sautner-style climate exposure, macro policy dictionaries) to produce a structured MacroSentimentResult with financial sentiment, policy stance, crisis signal, and climate-specific scores.

Ensemble Siblings

Custom Fine-Tuning

Fine-tune this model further on your own climate/ESG or financial labelled data:

pip install transformers datasets accelerate evaluate torch

python -m macro_sentiment.finetune \
    --data my_climate_labels.csv \
    --text-column text \
    --label-column sentiment \
    --base-model peyterho/climatebert-macro-sentiment \
    --output my-org/my-climate-model \
    --push-to-hub \
    --epochs 4 \
    --lr 1e-5 \
    --batch-size 32

The fine-tuning script (from the parent repo) accepts CSV, TSV, JSON, or JSONL. Labels can be strings ("positive", "negative", "neutral", "risk", "opportunity") or integers (0/1/2). Automatic label remapping handles ClimateBERT's non-standard label ordering.

Limitations

• —English only — the base ClimateBERT model was pre-trained exclusively on English climate text. Non-English climate disclosures should use the multilingual head in the full pipeline.
• —128-token max length — fine-tuned with 128-token truncation. Longer documents (e.g., full TCFD disclosures) should be chunked at paragraph level. The original ClimateBERT base was trained on paragraphs and may not perform as well on isolated sentences.
• —Climate data is small — only ~1,000 of the 20,034 training samples come from the climate sentiment dataset. The model's climate advantage comes primarily from pre-training, not from fine-tuning label supervision.
• —Label noise — the FiQA dataset uses continuous sentiment scores thresholded at ±0.15, introducing boundary noise. The climate_sentiment dataset uses annotated paragraphs which may contain mixed signals.
• —No temporal or entity awareness — treats each text independently. Cannot reason about whether "carbon prices rising" is positive or negative for a specific company or portfolio.
• —Smaller model trade-off — at 82M parameters (6 layers), this is the smallest head in the ensemble. It lags the 355M RoBERTa-Large by ~3 F1 points in-domain. For maximum accuracy, use the full pipeline in mode="all" to ensemble across all heads.

Citation

@techreport{bingler2023cheaptalk,
    title={How Cheap Talk in Climate Disclosures Relates to Climate Initiatives, Corporate Emissions, and Reputation Risk},
    author={Bingler, Julia and Kraus, Mathias and Leippold, Markus and Webersinke, Nicolas},
    type={Working paper},
    institution={Available at SSRN 3998435},
    year={2023}
}

@article{loughran2011liability,
    title={When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks},
    author={Loughran, Tim and McDonald, Bill},
    journal={The Journal of Finance},
    volume={66},
    number={1},
    pages={35--65},
    year={2011}
}

Framework Versions

• —Transformers 5.6.2
• —PyTorch 2.11.0+cu130
• —Datasets 4.8.4
• —Tokenizers 0.22.2

License

Apache 2.0