party2d/docs/union_mapping.md

Party Union Mapping

Individual Party Guarantee

Every party_id in the output CSV represents an individual political party, never an electoral alliance or bloc.

Electoral alliances and blocs are handled in one of two ways:

1. Decomposed via mean-constituent averaging (N=123 mappings): Shared manifesto data feeds into individual constituent party estimates. The output contains the constituents, not the alliance.
2. Excluded with documented justification (see "Excluded Alliance Labels" below): Alliance labels with no mappable constituents are dropped from the output.

A systematic audit of all output parties is provided by scripts/audit_party_types.py, which produces scripts/party_type_audit.csv classifying every party with evidence. Post-estimation verification in 02_post_estimation.jl hard-fails if any union/alliance PF ID appears in the output.

Overview

Expert surveys often rate individual constituent parties separately, while manifesto data is published under a union/coalition/merged party name. This document maps individual party PartyFacts IDs to the union PartyFacts IDs used in text_data.csv.

Direction: Individual party (expert data) → Union (manifesto/text data)

Model Integration (V4 Mean-Constituent Model)

The V4 Stan model (stan_model_2dim_v4.stan) uses these mappings to produce individual party estimates for all constituent parties. Instead of collapsing expert data onto the union ID, V4 gives every constituent its own latent position (theta) and links shared manifesto data to the mean of constituent thetas.

How it works

1. Each constituent gets its own segment and random walk. CDU (1375) and CSU (1731) each have independent theta trajectories. The union ID (211) gets NO segment.

2. Manifesto observations constrain the mean. A CDU/CSU manifesto in year t enters the likelihood as:

   pos = mean(theta[dim, rr_CDU_t], theta[dim, rr_CSU_t])
   

   The observation is counted once (no double-counting), but it pulls both constituents' thetas via their average.

3. Expert data constrains individuals directly. A CHES rating of CDU in 2019 maps to theta[dim, rr_CDU_2019] with no averaging. This is what identifies the difference between constituents.

4. Identification depends on data availability:

   • Periods with individual expert data (e.g., CHES 1999-2024): Constituent estimates separate meaningfully. CSU appears more conservative on galtan than CDU, matching known ground truth.
   • Periods without individual expert data (e.g., 1950s-1990s): Only the shared manifesto constrains the mean. The random walk prior pulls constituents toward similar values, so CDU ≈ CSU with wide credible intervals on the gap. The estimates gradually differentiate as expert data appears.

5. Backwards compatible. With an empty union_mapping.csv, all observations have n_const=1 and V4 reduces exactly to V3.

Data flow

R pipeline (00_data-management.R):
  - text_data keeps union PF IDs (211) for manifesto rows
  - expert/lr_data keeps individual PF IDs (1375, 1731)
  - Expert filtering: party in text_data OR party in constituent_parties

Julia pipeline:
  02_data_loading.jl  → loads union_mapping.csv → builds union_to_constituents dict
  03_data_preparation.jl → creates segments for CONSTITUENTS (not unions)
                         → builds flat arrays: n_const_man[], const_offset_man[], const_rr_man[]
  04_model_execution.jl  → passes constituent arrays to Stan

Stan model (stan_model_2dim_v4.stan):
  - Manifesto likelihood: averages theta over constituents per observation
  - Expert likelihoods: same averaging (nc=1 for individual obs = direct lookup)
  - No new parameters vs V3 — only data block and likelihood computation change

Output

Post-estimation produces individual party rows (CDU=1375 and CSU=1731 as separate rows) with a union_party_id column (= 211 for both, NA for standalone parties). The anchor party is CDU (1375) instead of CDU/CSU (211).

Scale impact (smoke test, post-1990 data)

• 471 parties → 483 segments, 461 with valid segments (vs ~449 without unions)
• 708 union manifesto observations with multi-constituent averaging (3.2% of 21,991 total)
• 23,427 total constituent entries in flat arrays
• 9,870 segment-year positions (R)
• No new model parameters (theta simply covers more segment-years)
• Negligible performance impact: nc=1 fast path for >96% of observations

Data quality (verified 2026-02-08)

• No double-counting: No constituents appear in text_data alongside their union
• Union IDs verified absent from output via post-estimation check in 02_post_estimation.jl
• No chain mappings: No PF ID serves as both union target and constituent of another union
• No duplicate rows in text_data after deduplication
• All flagged-for-review parties in audit script are confirmed individual parties (false positives from name patterns)

Detection Methodology

scripts/diagnose_party_mismatches.R uses a multi-signal approach, ranked by reliability:

Phase 1: MARPOR progtype variable (definitive)

The raw Manifesto Project data (MPDataset_MPDS2025a.csv) contains a progtype variable classifying each manifesto entry:

progtype	Meaning	Relevance
1	Party's own manifesto	Individual party, no union issue
2	Programme of 2+ parties (individual tracking)	Joint manifesto: each constituent gets its own CMP code with identical scores
3	Electoral manifesto by a single party	Individual party
4	Estimated from another party's programme	Party had no manifesto, inherited another party's scores
5	Average of member parties' manifestos	MARPOR computed average from constituent parties
6	Other	Miscellaneous
8	Party bloc programme	Bloc-level CMP code representing multiple parties
9	Non-standard text	Various sources

Phase 1a (progtype=2): Group by (country, date) where progtype=2 to find which CMP codes shared a joint manifesto at each election. Map each to PF IDs. If some are in text_data and others are expert-only, create constituent mappings.

Phase 1b (progtype=5): Average-of-members entries have their own CMP code. Find constituents via PartyFacts "composed of:" comments and manifesto party names.

Phase 1c (progtype=8): Bloc-level entries have their own CMP code. Find constituents via PartyFacts "composed of:" comments and manifesto party names.

Phase 1d (progtype=4): Proxy entries where MARPOR estimated from another party. Currently detects 0 expert-only parties with progtype=4.

Phase 2: PartyFacts metadata enrichment

Phase 2a (comment parsing): Search text_data party comments for "composed of:" patterns. Match mentioned abbreviations against expert-only parties using word-boundary matching.

Phase 2b (parlgov "+" notation): parlgov uses "+" to denote unions (e.g., "CDU+CSU", "CCD+CDU"). Parse fragments and match against expert-only parties.

Phase 2c (name fragment matching): For text_data parties with "/" or "-" in names, split into fragments and match against expert-only parties. Bilingual disambiguation: if all fragments resolve to the same PF ID, the "/" separates language variants (e.g., Swiss SPS/PSS), not constituents.

Phase 3: LLM verification (optional)

For remaining unmatched expert-only parties, uses gpt-4.1 via GESIS OpenWebUI with enriched prompts containing: all name_short values across datasets, year ranges, PartyFacts comments, progtype history, and the full list of text_data parties in the same country. Skip with --skip-llm flag.

Matching safeguards

• Word-boundary matching: Uses \b regex boundaries to prevent substring false positives (e.g., "DS" matching "NDSI")
• Minimum 2-character terms: Single-letter abbreviations (e.g., "K", "G") are excluded from matching
• Deduplication: When the same mapping is detected by multiple methods, the highest-reliability method is kept (manual > progtype > comments > parlgov > name_fragment > LLM)
• Idempotent: The script strips detection_method from existing mappings on re-run

How This Was Built

1. Original 36 mappings (2026-02-06/07): String matching + LLM + RA verification + independent research
2. Expanded with progtype (2026-02-07): Rewrote scripts/diagnose_party_mismatches.R to use MARPOR progtype (types 2, 4, 5, 8), PartyFacts comments, parlgov "+", and bilingual-aware name matching
3. LLM verification with gpt-4.1 (2026-02-07): Ran enriched prompts through gpt-4.1 via GESIS OpenWebUI for 345 remaining expert-only parties. Found 32 additional verified constituent mappings across 25 countries.
4. Bloc-centric LLM sweep (2026-02-07): For each of 38 unmapped progtype=8 bloc parties in text_data, sent targeted gpt-4.1 queries asking which expert-only parties are actual constituents. Found 17 new mappings (Brazil PT-led coalitions, Chilean Concertación, Serbian blocs, Latvian unions, etc.).
5. Manual research (2026-02-07): Verified remaining unmapped blocs. Added 4 manual mappings (PL: .N→KO; CL: PC→Concertación, PH→Concertación; ES: Amaiur→EH Bildu). Confirmed remaining 27 blocs have no mappable expert-only constituents (constituents either already in text_data, not in expert surveys, or in fluid coalitions).
6. Temporary election coalitions (2026-02-08): Systematic analysis of progtype=2 joint manifesto groups. Grouped by identical CMP content (all 142 per* columns) to separate left and right coalitions within the same election. Found 9 unmapped expert-only parties that shared manifestos with text_data parties (Italy 2001/2006/2013, France 2017). Added CCD→CdL from orphan analysis (1 manual). Total: 10 new mappings.
7. Individual party guarantee cleanup (2026-02-08): Removed 11 progtype_2/progtype_2_joint entries that mapped real individual parties as unions of zero-data coalition partners. Removed 2 chain mappings (union→union→constituent). Added 4 new mappings for genuine alliances (MX: Salvemos a México→{PRI, PVEM}; IL: Joint List→{Hadash, Balad}). Documented classification decisions for all 15 dual-progtype parties and 4 pure bloc labels. Added post-estimation verification check and audit script.

Mapping Table

The canonical mapping is in data/union_mapping.csv with columns:

• manifesto_pf_id — PartyFacts ID of union (in text_data)
• manifesto_name — Union display name
• expert_pf_id — PartyFacts ID of individual constituent (in expert_raw)
• expert_name — Individual party display name
• country — ISO2 country code
• relationship — Description of the relationship
• status — implemented (active in pipeline) or pending (awaiting implementation)
• detection_method — How detected: manual, progtype_2, progtype_2_joint, progtype_5, progtype_8, progtype_4, composed_comment, parlgov_plus, name_fragment, llm_verified

Detection Method Breakdown

Method	Count	Description
llm_verified	48	gpt-4.1 verified (party-centric + bloc-centric)
manual	45	Hand-verified mappings (includes MX/IL bloc mappings, CCD→CdL)
progtype_8	23	Bloc-level CMP entries matched to expert-only constituents
name_fragment	4	"/" or "-" name splitting matched to expert-only parties
composed_comment	2	PartyFacts "composed of:" comments
progtype_5	1	Average-of-members entries (Croatia ZL)

Total: 123 rows mapping 116 unique expert parties to 87 union parties across 41 countries

Bloc Coverage

Of 49 progtype=8 bloc parties in text_data, 22 (44.9%) have at least one constituent mapped. The remaining 27 have no mappable expert-only constituents because:

• Actual constituents already have their own text_data entries (e.g., KDNP, Yesh Atid, TB-LNNK)
• Actual constituents are not in expert surveys (e.g., Hadash, Ra'am, Balad, VS, DKP)
• Coalition membership is too fluid for static mapping (Panama, Colombia, Israel shifting coalitions)
• The entry is a single party coded as a bloc (e.g., German Minority, Red-Green Unity List)

Temporary Election Coalitions (progtype=2) — REMOVED

The progtype=2 joint manifesto mappings (Italy 2001/2006/2013, France 2017, Belgium 1971) were removed in the 2026-02-08 cleanup because they mapped real individual parties as "unions" of tiny coalition partners with zero data. See "Removed Mappings" below for details.

The CCD(1767)→CdL(6241) mapping from orphan analysis remains, as CCD was a genuine long-term CdL constituent (progtype=1 in 1996, part of CdL bloc from 2001).

Unmappable Expert-Only Parties

These parties appear in expert surveys but have no manifesto union to map to. Verified against text_data_unfiltered.csv (pre-temporal-filter, 1238 parties) on 2026-02-07.

Filter retest results: 54 of the 80 original RA task parties have manifesto data in the unfiltered text_data, but all were dropped by the temporal continuity filter (requires ≥3 years with gaps ≤6). These parties have their own CMP data but too few observations. They are correctly handled via union mapping (where applicable) or excluded (where standalone). See scripts/party_mismatch_ra_task.csv for full year-level detail.

Expert PF ID	Name	Country	Manifesto years (unfiltered)	Expert years	Why unmappable
5623	Compromís	ES	0 (CMP codes map to different PF IDs)	2 (2018, 2023)	No manifesto data under PF ID 5623. CMP codes 33098/33093/33914 are separate PartyFacts entries.
4363	FDG (Front de Gauche)	FR	1 (2012)	1 (2014)	1 CMP year only. PCF (1251) already has its own expert data.
5731	NNP	ZA	0	1 (1999)	Apartheid party that dissolved into ANC 2005. Ideologically incompatible mapping.
5553	FREPASO	AR	1 (1995)	4 (1995-2001)	1 CMP year only. Independent party; Alianza (1999) was separate CMP entity.
8122	CF (Consenso Federal)	AR	1 (2019)	2 (2019-2020)	1 CMP year only. One-off coalition.
4182	FPL+UCeDé	AR	1 (2003)	3 (1987-1991)	1 CMP year only. Incoherent entity (UCeDé 1987 ≠ FPL coalition 2003).
6160	FR (Frente Renovador)	AR	0	4 (2013-2019)	No CMP code at all. Major ideological shift 2013→2019.
5879	CD (Centro Democrático)	CO	1 (2014)	3 (2014-2019)	1 CMP year only. Uribe's party, independent.
4411	PNI/PPN	CR	0	1 (1974)	No CMP code. Defunct 1970s party.
7412	EK/DEK	CY	0	1 (1970)	No CMP code. Defunct 1970 far-right party.
3935	United Opposition	GE	1 (2008)	1 (2008)	1 CMP year only. Was anti-UNM alliance (not led by UNM).

Rejected Mappings

These were proposed by the RA but found incorrect during independent research:

Individual PF ID	Name	Proposed target	Why wrong
5623	CC/Compromís	81 (CCa-PNC-NC)	Name coincidence. Compromís is a Valencian left party; CCa is a Canarian right party.
4363	FDG	1251 (PCF)	FDG was a multi-party alliance (PCF + Parti de Gauche). PCF already has its own 2014 expert data. Mapping FDG→PCF would double-count.
5731	NNP	1219 (ANC)	The NNP (ex-apartheid National Party) dissolved into ANC in 2005 as political capitulation. NNP positions are diametrically opposed to ANC on every dimension.

Completeness Audit (2026-02-08)

Systematic analysis of 631 orphan expert parties (with 4+ observations) across 46 countries (excluding FPTP systems like UK, US, Canada where party unions are not a meaningful concept). Cross-referenced with MARPOR progtype data and PartyFacts metadata.

Methodology

1. Progtype=2 sweep: Identified all 93 MARPOR entries with progtype=2. Grouped into 27 content-identical subgroups. Found 12 expert-only parties with progtype=2 data; 10 were mappable (mapped above), 2 had no matching text_data party in their coalition.

2. Orphan analysis: For each of 46 countries, identified expert-only parties not in text_data and not already mapped. Examined PartyFacts metadata (names, comments, ideology tags) and MARPOR progtype history for each.

3. Coalition verification: For Italian orphans specifically, checked each party's MARPOR entries for progtype=2/8 membership and ideological alignment with existing text_data coalition parties.

Major standalone orphan parties (not coalition members)

These are significant parties with substantial expert data but no manifesto/union data. They are genuinely standalone — not missed union constituents.

Country	Party	PF ID	Expert obs	Why standalone
AT	BZÖ	599	7	Splinter from FPÖ; own CMP data filtered out
BE	PTB/PVDA	1753	14	Independent far-left; never part of any coalition
CZ	ANO	2141	13	Babiš party; own CMP data exists but too recent
CZ	Piráti	2047	10	Standalone; own CMP data exists
FR	REM/R	5857	10	Macron's party; too new for sufficient CMP data
FR	FI	5858	8	Mélenchon's party; standalone
IT	M5S	2046	13	Five Star Movement; progtype=1 only, standalone
IT	FDI	2280	10	Fratelli d'Italia; progtype=1 only, standalone
RO	USD	120	23	Social democratic bloc; own CMP code exists
SK	OĽaNO	2130	13	Populist party; standalone
CO	PCC	1577	19	Conservative party; own CMP code exists but filtered
BR	PSDB	225	13	Social democrats; standalone (not PT coalition)

Inactive union mappings

These union targets exist in union_mapping.csv but are NOT in text_data, making their constituent mappings inactive:

Union PF ID	Name	Country	Constituents	Why inactive
1212	SEL	IT	SEL(7031)	1212 not in text_data
1737	Olive Tree	IT	PCI(34), DS(878)	1737 not in text_data; only 2 MARPOR years

Note: 962 (CCD+CDU→CDU-Italy) was removed from the mapping entirely because it created a chain (962 is both a constituent of UdC and a union target). See "Removed Mappings" below.

Conclusion

The 123 mappings comprehensively cover: (1) all permanent unions with separate expert data, (2) all progtype=8 bloc parties with mappable expert-only constituents, and (3) CCD as an additional Italian coalition member identified through orphan analysis. Remaining orphan expert parties are genuinely standalone parties whose manifesto data was either filtered out by temporal continuity requirements or does not exist.

Removed Mappings (2026-02-08)

11 progtype=2 joint manifesto entries removed

These entries mapped real individual parties (with substantial text and expert data) as "unions" of temporary coalition partners that had zero text data AND zero expert data. Every constituent had no information to contribute, making the mapping harmful (it reduced real parties to averages with phantom partners).

Union PF ID	Union Name	Constituent PF ID	Constituent	Country	Why removed
8054	DS	279	M-DL	IT	M-DL has 0 text, 0 expert data; DS has 120 text, 3 expert
1404	PRC	1635	PdCI	IT	PdCI has 0 text, 0 expert data; PRC has 58 text, 15 expert
1404	PRC	1711	RnP	IT	RnP has 0 text, 0 expert data
6241	CdL	888	NPSI	IT	NPSI has 0 text, 0 expert data; CdL's progtype_8 mappings (AN, CeD, FI) remain
6241	CdL	2415	ALD	IT	ALD has 0 text, 0 expert data
768	IdV	115	P-UDEUR	IT	P-UDEUR has 0 text, 0 expert data; IdV has 26 text, 8 expert
768	IdV	1369	SVP	IT	SVP has 0 text, 0 expert data (under this PF ID)
1221	Lega	365	PdL	IT	PdL has 0 text, 0 expert data; Lega has 77 text, 24 expert
1595	UMP	3229	UDI	FR	UDI has 0 text, 0 expert data; UMP has 66 text, 18 expert
49	openVLD	622	CD&V	BE	CD&V is a large party (own PF ID); mapping created false dependency
554	PRL	622	CD&V	BE	Same issue: CD&V already has its own data pipeline

2 chain mapping entries removed

These created chain dependencies (union A → union B → constituents), which the pipeline does not support:

Union PF ID	Union Name	Constituent PF ID	Constituent	Country	Why removed
962	CCD+CDU	763	CDU (Italy)	IT	962 is itself a constituent of UdC (201). CDU-Italy (763) has 0 data. Chain: 201→962→763.
5939	PàF	1742	AD	PT	5939 is itself a constituent of CDS-PP (1308). AD (1742) has 0 data. Chain: 1308→5939→1742.

New Mappings Added (2026-02-08)

Union PF ID	Union Name	Constituent PF ID	Constituent	Country	Rationale
3979	Salvemos a México	1474	PRI	MX	PRI-PVEM electoral coalition (2006-2012); PRI has extensive text + expert data
3979	Salvemos a México	446	PVEM	MX	PVEM is second constituent; has its own text + expert data
7912	Joint List	421	Hadash	IL	Arab party coalition (2015-2021); Hadash has CHES 2022 data
7912	Joint List	1663	Balad	IL	Balad is constituent; has CHES 2022 data

Excluded Alliance Labels

These party IDs appear in text_data as bloc/alliance labels but are NOT decomposed via union mapping because no constituent has data in the pipeline:

PF ID	Name	Country	Text data	Expert data	Why excluded
3995	Alianza Acción Opositora	PA	48 obs	0	No expert survey coverage for Panama. No constituents identifiable in pipeline.

Note: Several other progtype=8 bloc parties in text_data also have no mapped constituents (see "Bloc Coverage" below), but they remain in the output either because (a) they have their own expert data (e.g., 2988 Georgian Dream, 3916 Alianza Grande) or (b) they function as individual parties despite bloc coding (e.g., 1527 Enhedslisten, 1439 German Minority).

Classification Decisions

Dual-progtype parties (both progtype=1 and progtype=8)

These 15 parties have MARPOR entries under both individual (progtype=1/3) and bloc (progtype=8) codes. Each was individually reviewed.

Classified as individual parties (no action needed):

PF ID	Name	Country	Evidence
57	SLD	PL	Dominant Polish left party; bloc coding reflects coalition leadership, not alliance status
81	CCa	ES	Canarian regionalist party; single party with local coalition leadership
1056	SC	LV	Latvian party; dual coding reflects different election formats
1150	SDE	EE	Estonian Social Democrats; individual party
1396	Samfylkingin	IS	Icelandic Social Democratic Alliance; merged into single party
1439	MN	PL	German Minority in Poland; single ethnic party coded as bloc
1527	Enhedslisten	DK	Red-Green Alliance; functions as single party since 1989
1691	FiDeSz-KDNP	HU	FiDeSz dominant; KDNP (1412) has separate PF ID and data
2172	ENM	GE	United National Movement; single party with bloc-era coding
2228	BYuT	UA	Tymoshenko bloc; functions as single Ukrainian party
2252	Yabloko	RU	Russian liberal party; individual entity

Classified as individual parties after research:

PF ID	Name	Country	Decision	Evidence
506	VL-TB/LNNK	LV	Individual (merger party)	National Alliance formed 2010 by merger of VL and TB/LNNK. Post-merger, functions as single party. MARPOR data 2010-2022. TB/LNNK (1704) has separate pre-merger data (1998-2014). Not mapped as union because 1704 is already a union target with its own constituents; mapping would create a chain.
1586	sp.a-SPIRIT	BE	Already handled	Already mapped as constituent of sp.a (1680) in union_mapping.csv. 0 text, 0 expert data under this PF ID.
7599	Kahol Lavan	IL	Individual party	Short-lived centrist party (2019-2020). Has own expert data (CHES 2021, V-Party 2019). Unified entity, not a multi-party alliance.
2988	Georgian Dream	GE	Individual party	Despite progtype=8 coding, functions as a single party-movement. Has own expert data (GPS 2019, V-Party 2012/2016).
3916	Alianza Grande	CO	Individual party (catch-all PF ID)	PF ID covers multiple Colombian coalitions. Has own expert data (CHES 2020). No separate constituent expert data exists.

Classified as alliance and mapped:

PF ID	Name	Country	Decision	Evidence
7912	Joint List	IL	Alliance → mapped to Hadash (421), Balad (1663)	Arab party coalition (2015-2021). GPS explicitly names 4 constituents. Hadash and Balad have separate CHES 2022 data and their own text_data.
3979	Salvemos a México	MX	Alliance → mapped to PRI (1474), PVEM (446)	PRI-PVEM electoral coalition. Both constituents have extensive text and expert data.

Audit Methodology

The audit script scripts/audit_party_types.py systematically checks every party in the output CSV:

1. Union mapping check: Verifies no manifesto_pf_id from union_mapping.csv appears in output (hard fail).
2. Constituent check: Identifies parties that are expert_pf_id in the mapping (expected: these are individual constituents of unions).
3. Expert data check: Flags parties with no expert survey data (text-only entities).
4. Name pattern check: Scans PartyFacts names for alliance indicators (keywords: alliance, coalition, bloc, front, union, alianza, frente; characters: +, /, &).
5. Classification: Each party gets one of: individual_party, flagged_for_review, error_union_in_output.

To re-run after data updates:

python3 scripts/audit_party_types.py

Output: scripts/party_type_audit.csv with columns: party_id, name, country, in_union_mapping_as_union, in_union_mapping_as_constituent, has_expert_data, name_flags, classification, evidence.

Post-estimation verification (02_post_estimation.jl): After extracting estimates, loads all manifesto_pf_id values from union_mapping.csv and checks none appear in the output party_id column. If any do, the script errors with a hard fail.