party2d/docs/DATA_CODING_PRINCIPLES.md

# Data Coding Principles for 4D Latent Trait Model

## V4 Implementation (Current Version)

**As of V4 (2025-11-18), manifesto items implement the bipolar bridge structure described in this document.**

**Key Changes from V3.x**:
- ✅ Manifesto items now load on TWO dimensions (bipolar bridges)
- ✅ Data format: `type_high` and `type_low` columns replace single `type`
- ✅ Stan model: Unified `Gamma_man` matrix replaces per-dimension arrays
- ✅ Measurement consistency: Manifesto items match expert data structure

**Why V4?**
- Better identification (each observation informs two dimensions)
- Estimated correlations (not imposed by construction)
- No double-counting (each quasi-sentence counted once)

See CHANGELOG.md for full V4 migration details.

---

## Model Structure Overview

The model estimates **four unipolar latent dimensions**:
- **pro_market**: Support for market liberalization
- **pro_welfare**: Support for welfare state expansion
- **cosmopolitan**: Support for internationalism, diversity, openness
- **traditional**: Support for nationalism, security, traditional values

These are **separate dimensions**, not two bipolar scales. Correlations between dimensions (e.g., cosmopolitan-traditional) are **estimated empirically**, not imposed by construction.

---

## Item Types and Loading Structure

### 1. Bipolar Bridge Items

**Definition**: Items where the sample includes mentions of BOTH sides of an issue, and "positive" counts mentions favoring one pole.

**Structure**:
- `sample` = mentions of issue (any direction)
- `positive` = mentions favoring one pole
- `positive/sample` ratio varies from 0 to 1

**Loading**: Should load on **ONE dimension only**

**Examples**:

**Manifesto Data**:
```
var: "Multiculturalism"
type: "cosmopolitan"
sample: per607 (pro-multiculturalism) + per608 (anti-multiculturalism)
positive: per607 (pro-multiculturalism)
```
- High ratio → high cosmopolitan (party favors multiculturalism)
- Low ratio → low cosmopolitan (party opposes multiculturalism)
- Anti-multiculturalism is **implicitly measured** as (sample - positive)

**PolDem Data**:
```
var: "Immigration (Media)"
type: "cosmopolitan"
sample: all immigration mentions (direction != 0)
positive: pro-immigration mentions (direction > 0)
```
- High ratio → high cosmopolitan (media coverage shows party supporting immigration)
- Low ratio → low cosmopolitan (media coverage shows party opposing immigration)

### 2. Why One Loading Suffices for Bipolar Items

**Question**: Shouldn't anti-immigration also load on traditional?

**Answer**: No, because:

1. **Both poles are already captured**: The bipolar structure means low cosmopolitan (anti-immigration) is automatically measured
2. **Avoids double-counting**: Each mention/quasi-sentence contributes to exactly ONE item
3. **Empirical correlations emerge naturally**: If anti-immigration parties also score high on nationalism/law-and-order, the **posterior correlation** between cosmopolitan and traditional will reflect this
4. **More flexible model**: Cosmopolitan-traditional relationship is **estimated**, not imposed

**Imposed vs. Estimated Correlation**:
- If we double-load immigration on both cosmopolitan (negative) and traditional (positive), we **force** them to be opposites
- By loading only on cosmopolitan, we let the data reveal whether anti-immigration parties are also nationalist (empirical question)

---

## Coding Decision Rules

### Rule 1: Each Manifesto Code Appears in ONE Item Only

**Good** (current structure):
```
"Multiculturalism" (cosmopolitan):
  - per607 (Positive), per608 (Negative)

"National Identity" (traditional):
  - per601 (Positive), per107 (Negative)
```
- per607/per608 only in cosmopolitan
- per601/per107 only in traditional
- Correlation between dimensions is empirical

**Bad** (double-loading):
```
"Multiculturalism" (cosmopolitan):
  - per607 (Positive), per601 (Negative)

"National Identity" (traditional):
  - per601 (Positive), per607 (Negative)
```
- per601 and per607 counted twice
- Imposes perfect negative correlation between cosmopolitan/traditional

### Rule 2: Stance Assignment Within Items

Within each item (var), codes are assigned stance based on:
- **Positive**: Codes indicating support for the item's construct
- **Negative**: Codes indicating opposition to the item's construct

**Example - "Internationalism" (cosmopolitan)**:
- per107 (Internationalism positive): stance = "Positive"
- per109 (Internationalism negative): stance = "Negative"
- Result: High per107 / low per109 → high cosmopolitan score

### Rule 3: PolDem Direction Mapping

PolDem uses `direction` variable (-1, 0, +1):
- `direction > 0`: Support for the issue as coded
- `direction < 0`: Opposition to the issue
- `direction == 0`: Ambivalent (exclude from analysis)

**Aggregation**:
```r
poldem_processed %>%
  filter(direction != 0) %>%  # exclude neutral
  group_by(party, year, country, issue_cat) %>%
  summarise(
    sample = n(),  # all non-neutral mentions
    positive = sum(direction > 0)  # supportive mentions only
  )
```

---

## Special Cases

### Immigration (Direction Ambiguity)

**Codebook says**: "Opposition to restrictive immigration"

**Interpretation needed**: Does `direction = +1` mean:
- A) Support for "opposition to restrictions" → pro-immigration (cosmopolitan)
- B) Support for "restrictions" → anti-immigration (traditional)

**Resolution**: Must manually inspect sample sentences before finalizing coding.

If interpretation A is correct:
```r
issue_cat == "immig" & direction > 0  → positive for cosmopolitan
issue_cat == "immig" & direction < 0  → negative for cosmopolitan
```

If interpretation B is correct:
```r
issue_cat == "immig" & direction > 0  → negative for cosmopolitan
issue_cat == "immig" & direction < 0  → positive for cosmopolitan
# (REVERSED)
```

### Europe/Euro Items

EU integration naturally maps to cosmopolitan-traditional dimension:

**Manifesto Data**:
- Add new items using per108 (EU integration positive) and per106 (EU integration negative)
- Create separate vars: "EU Integration Support" (cosmopolitan), "Euroskepticism" (traditional)

**PolDem Data**:
```r
"EU Integration Support (Media)" (cosmopolitan):
  issue_cat = "europe" or "euro"
  sample = all mentions
  positive = direction > 0 (pro-EU)

"Euroskepticism (Media)" (traditional):
  issue_cat = "europe" or "euro"
  sample = all mentions
  positive = direction < 0 (anti-EU)
```

**Note**: Same sentences contribute to BOTH items, but counting opposite directions. This creates natural negative correlation between cosmopolitan/traditional.

**Alternative approach** (cleaner, recommended): Load only on cosmopolitan:
```r
"EU Position (Media)" (cosmopolitan):
  issue_cat = "europe" or "euro"
  sample = all mentions
  positive = direction > 0
```
This is sufficient if we treat EU as a bipolar cosmopolitan item.

---

## Data Structure Requirements

### Manifesto Data Format (party-year-var level)

Each row represents one item for one party-year:

| party | country | year | var | type | sample | positive | project |
|-------|---------|------|-----|------|--------|----------|---------|
| 211 | DE | 2013 | Multiculturalism | cosmopolitan | 45 | 23 | Manifesto |
| 211 | DE | 2013 | National Identity | traditional | 67 | 58 | Manifesto |
| 211 | DE | 2013 | Economic Intervention | pro_welfare | 102 | 78 | Manifesto |

- **var**: Item name (e.g., "Multiculturalism", "Economic Intervention")
- **type**: Dimension it loads on (pro_market, pro_welfare, cosmopolitan, traditional)
- **sample**: Total quasi-sentences mentioning this issue
- **positive**: Quasi-sentences with positive stance toward this item

### PolDem Data Format (same structure)

| party | country | year | var | type | sample | positive | project |
|-------|---------|------|-----|------|--------|----------|---------|
| 211 | DE | 2013 | Immigration (Media) | cosmopolitan | 23 | 8 | PolDem |
| 211 | DE | 2013 | Nationalism (Media) | traditional | 15 | 12 | PolDem |

Combined using `bind_rows()` to create unified dataset.

---

## Summary

1. **Bipolar items load on one dimension only** - the ratio captures both poles
2. **Each manifesto code appears in exactly one item** - no double-counting
3. **Correlations between dimensions are estimated, not imposed** - more flexible model
4. **Direction reversals are handled within items** - via stance assignment (Manifesto) or direction sign (PolDem)
5. **All items must allow varying positive/sample ratios** - mix of positive and negative stances required

This structure preserves the conceptual independence of the four dimensions while allowing the data to reveal their empirical relationships.