nycuriosity/data

Methodology & data notes

Mission, vision, inclusion criteria, taxonomy, and technical notes on how the Directory, Connect, and Affinity Network views are built. The Connect directory is manually curated by the SCE team. All information is kept private unless requested to be shared.

What We're Building

A queryable, interactive tool surfacing organizations and people working at the intersection of technology and government effectiveness.

Mission

To map and connect the ecosystem of organizations building government capacity, making it easier to find, learn from, and collaborate with the people and institutions doing this work.

Vision

Good policy fails because the practitioners, technologists, researchers, and funders working to fix government are operating in silos, without a clear view of each other and with few pathways to collaborate. Great talent is left out not by intent, but because it cannot find its way in.

We believe visibility is infrastructure. A clear map of who is doing what is the first step toward state capacity acting like an ecosystem, not a collection of isolated players.

We're building that map. And we're going further: adding tools so people can find funders, collaborators, and partners who can actually move their work forward. And launching events to make sure those connections happen in real life, not just on a screen.

What "State Capacity" Means Here

Organizations that help government become more capable, not organizations that do government's work for it. An org is added if it helps the American government expand its own effectiveness, capacity, or legitimacy.

Data Fields

  • Org Name — the name of the organization.
  • Primary Segment — primary categorization based on the org's focus.
  • Secondary Segments — additional areas the org works on.
  • Geography — geographic level at which the org operates.
  • Description — a 1–3 sentence summary of what the org does.
  • Funding Model — how the organization is funded.
  • Funding Detail — free-text description of funding detail.
  • Website — the org's homepage.
  • Problem Area — problem topics classified into 7 buckets: Participatory Democracy, Procurement & Operations, Service Delivery, Talent & Hiring, Technology & Data, Test & Learn, Domains.
  • Problem Topic — 36 specific issues nested under the Problem Areas.

Segment Taxonomy

An organization's segment describes its primary operating model.

  • Advocacy — Organizations pushing for policy and structural changes that enable better government.
  • Capacity Building — Groups training and strengthening the government workforce.
  • Community — Networks and communities of practice connecting government practitioners.
  • Digital Services & Consulting — Firms doing hands-on technology and service design work with government.
  • Ecosystems — Connectors and conveners building the field itself.
  • Fellowships — Programs placing talent inside government agencies.
  • GovTech — Companies building software and tools for government customers.
  • Government — Public agencies and internal teams leading change from within.
  • Investor — Funds and investors backing companies in this space.
  • Philanthropy — Foundations and funders supporting the broader ecosystem.
  • Research — Organizations studying what works and building the evidence base.

Problems Taxonomy

An org earns a tag if it directly solves the problem for government customers, funds or builds the infrastructure that moves the needle on it, or sets the standards that govern it. The problem needs to be a focus for the org; not addressed by ancillary work or with one-time funding, truly part of the mission and vision.

Problem AreaProblem TopicDefinition
Participatory DemocracyCivic EngagementConnecting residents to government and strengthening democratic participation.
Democracy InfrastructureStrengthening the structural foundations of democratic governance and institutions.
Transparency & AccountabilityTransparency, open data, FOIA, and public accountability infrastructure.
Procurement & OperationsOperational ExcellenceImproving how government operates internally — processes, systems, and workforce efficiency.
Procurement ReformModernizing how government buys, evaluates vendors, and manages technology and service contracts.
Reaction TimeImproving the speed at which government can respond to problems and opportunities.
Regulatory & Administrative BurdenReducing unnecessary complexity in government rules, forms, procedures, and compliance requirements.
Service DeliveryBenefits AccessImproving resident access to and delivery of government benefit programs.
Service DesignHuman-centered design of government services, forms, processes, and digital interfaces.
Talent & HiringExpert ContributionPlacing or mobilizing subject matter experts to work on government problems.
Hiring ArchitectureRedesigning how government structures its talent acquisition and retention systems.
Hiring ProcessImproving the mechanics of government recruitment, assessment, and onboarding.
Talent PipelineBuilding pathways for technologists, policy experts, and public servants to enter government.
Workforce DevelopmentUpskilling existing government workers and the broader labor force served by government programs.
Technology & DataData IntegrationBreaking down fragmented data systems that prevent agencies from sharing information.
Data SecurityProtecting government systems and data from cyber threats.
Legacy SystemsReplacing or modernizing outdated government technology infrastructure.
Shared PlatformsBuilding reusable, open-source, or collectively-owned technology infrastructure for government.
Test & LearnEvidence UseEmbedding research and data into government decision-making and program design.
Feedback LoopsBuilding mechanisms for government to collect and act on resident and stakeholder input.
Iterative LearningBuilding government capacity to test, learn, and improve through iteration.
Outcomes MeasurementUsing data and evidence to track and improve government program results.
Scaling What WorksIdentifying proven approaches and systematically replicating them across jurisdictions.
DomainsAI in GovernmentGovernment adoption, governance, and responsible deployment of artificial intelligence.
Broadband & Internet ConnectivityExpanding government capacity to deploy and manage broadband infrastructure.
Child WelfareTechnology and reform capacity for child protective services, foster care, and family systems.
Criminal JusticeTechnology and reform capacity for law enforcement, courts, corrections, and reentry.
Crisis ResponseGovernment capacity for emergency management, disaster response, and 911 systems.
Economic MobilityGovernment programs and systems that improve economic opportunity for residents.
Education DeliveryTechnology capacity for government education agencies to deliver and improve learning.
Healthcare AccessTechnology capacity for government health programs — Medicaid, Medicare, public health delivery.
Housing & Land UseGovernment capacity for housing policy, permitting, zoning, and affordable housing delivery.
Permitting & LicensingModernizing government permitting, licensing, and code enforcement processes.
Physical InfrastructureTechnology capacity for managing roads, utilities, transit, and built environment.
Public HealthCapacity of public health departments and agencies to deliver and modernize health programs.
Tax & RevenueTechnology capacity for government finance, tax collection, and fiscal management.

The directory

The directory is a straight rendering of the underlying data. It supports faceted filters and a semantic search box:

  • Search accepts either an org name or a natural-language phrase ("procurement reform in NYC", "AI in benefits delivery"). The query is tokenized, scored against a precomputed TF-IDF index of every org's description and tags, and results are returned ranked by relevance.
  • Primary segment is a multi-select on the org's main category.
  • Geography is a multi-select; an org matches if it operates at any of the selected levels.
  • Problem area and Problem topic are multi-selects on the curated taxonomy; an org matches if it carries any of the selected tags.

Connect

The Connect page is a directory of individuals and open opportunities — practitioners, researchers, funders, and others who have opted in to be findable, alongside open calls (grants, fellowships, jobs, collaborative projects) indexed by problem and offering type. It is separate from the org Directory: orgs belong in the Directory; people and opportunities belong in Connect.

Each entry carries: name, organization affiliation, role, offering, problem area(s), problem topic(s), geography, due-by date, details, and a contact preference (direct contact info, or a facilitated introduction through the curator). Entries are submitted through the on-page form and reviewed before going live.

Semantic search

The directory's search box and the affinity network share a precomputed search index. (The Search page uses structured token-chip filters rather than free text — described in the section above.) At build time, every org's description, problem-statement tags, segments, and funding-detail text are tokenized into a unified bag and a TF-IDF vector is stored. At query time, the browser tokenizes your query the same way, builds an IDF-weighted query vector, and computes cosine similarity against every org. Exact substring matches in the org name receive a 0.5 boost so typing an exact name returns it first. The full index ships as a static JSON file (~190 KB) — no server, no embedding API, no per-query cost.

This is significantly more than keyword matching: a query like "where should I work on procurement in NYC?" surfaces orgs tagged with Procurement Reform plus orgs operating in NYC plus orgs whose descriptions emphasize vendor evaluation, even when no single word matches everywhere. It is also significantly less than a true semantic embedding — it can't infer that "permits" and "licensing" refer to the same concept unless those words co-occur frequently in the corpus. For the size and vocabulary of this dataset, TF-IDF is the right cost/quality trade-off.

The affinity score

The network view draws an edge between two organizations when they appear to be working on the same underlying problems. The score is a weighted composite of four signals:

score = 0.40 × description similarity
+ 0.30 × shared problem statements
+ 0.15 × shared named funders
+ 0.15 × shared segments
Description 40%
The substance of what the org does (TF-IDF cosine).
Problems 30%
The specific issues each org is trying to fix.
Funders 15%
Theory of change: who believes in this approach.
Segments 15%
The mechanism: how the org operates.

The framing behind these weights: every signal points at the same underlying question, which is what is this org trying to accomplish, and who else is trying to accomplish the same thing? Problem statements tell you what they're working on. Descriptions tell you the substance. Funders tell you the theory of change. Segments tell you the mechanism.

Segment overlap is intentionally the smallest signal. An earlier version of the score gave segments 35% weight plus a primary-segment match boost — but that made the network show mostly the obvious thing (Research orgs cluster with Research orgs). The current weighting surfaces cross-segment surprises: a Research shop and a GovTech vendor that turn out to be working on the same problem with overlapping funders are now plausibly connected even though they live in different segment clusters.

Description similarity (40%)

The strongest signal, because the description text is where org-specific language lives: phrases like "permitting reform," "legislative capacity," "procurement modernization," or "AI policy." We compute a TF-IDF representation of each org's description, funding-detail text, segment names, problem-topic tags, and problem-area tags combined, then take the cosine similarity between every pair of vectors. Folding problem areas into this bag lets a free-text query like "procurement" match orgs whose description never uses the word but whose tags do.

TF-IDF (term frequency–inverse document frequency) downweights words that appear everywhere ("government," "technology," "policy") and upweights words that are distinctive. Two orgs that both talk about "procurement" will score higher than two orgs that both talk about "policy." Common words and short tokens are dropped before vectorizing.

Shared problem statements (30%)

The Jaccard similarity of the two orgs' problem-topic tag sets. Every org carries one or more tags from a curated list of 36; an org tagged with both Procurement Reform and Service Design shares a 50% Jaccard with one tagged with only Procurement Reform. Because the tags are curator-assigned, this is the highest-confidence signal in the dataset — when two orgs share problem topics, they really are working on the same thing. (The coarser Problem Area bucket is folded into the description-similarity signal but not into this Jaccard, since orgs share an Area too easily to be a high-confidence match.)

Shared funders (15%)

Funders are pulled from the free-text Funding Detail by string-matching against a curated list of ~50 well-known philanthropies, foundations, and venture funds (Schmidt Futures, Open Philanthropy, Omidyar Network, Ford, MacArthur, Knight, Bloomberg, Arnold Ventures, 8VC, a16z, etc.). For two orgs with at least one named funder each, we compute the Jaccard similarity of their funder sets. If neither has an extractable named funder, we fall back to a small bonus when their funding-model field matches exactly.

Limitation worth flagging. Approximately 64 of 329 orgs have at least one named funder detected. The curated list is not exhaustive, and many smaller foundations, family offices, and government grants are missed. Orgs that share an unlisted funder will not light up on this signal — the description and problem-statement scores carry them.

Shared segments (15%)

Plain Jaccard over the union of each org's primary and secondary segment tags. No primary-segment boost. Earlier versions added a +0.5 boost for matching primary segments, which made the graph collapse into same-segment cliques. Removing the boost and reducing the overall weight means segment overlap acts as a tiebreaker, not a dominant signal.

What gets dropped, what gets kept

Pairs with a composite score below 0.05 are discarded entirely. From the remaining pairs we apply two further filters:

  • Absolute floor: edges below 0.10 are not included in the dataset.
  • Per-node degree cap: we walk through the remaining edges in descending score order and keep an edge only if at least one of its two endpoints currently has fewer than 8 kept neighbors. This prevents central hubs from collecting hundreds of weak edges that produce a hairball, while still letting them be the most connected nodes.

The published dataset has 329 nodes and 1,763 edges, with a maximum edge score of 0.82 and a median of 0.10. The network view starts at a visual threshold of 0.18 for legibility; the slider exposes the full 0.10–0.40 range. A thicker, more saturated edge is a higher composite score (stronger affinity); a thin faint edge is a weaker match.

Inside the score table

For reference, here is what each signal can range over:

SignalWeightRangeNotes
TF-IDF cosine (descriptions, problem topics + areas, segments, funding text)0.400 to ~0.6 in practiceRarely exceeds 0.5 because descriptions are short.
Problem-statement Jaccard0.300 to 1.0Highest-confidence signal — curator-assigned tags.
Named-funder Jaccard0.150 to 1.0Falls back to 0.15 if funding models match and no funders are detected.
Segment Jaccard (no primary boost)0.150 to 1.0Intentionally a tiebreaker, not a dominant signal.

What the graph does and does not show

It shows structural affinity: which orgs are working on the same underlying problems, talk about them in similar ways, and often share funders. The strongest edges in the dataset are between orgs that any expert in the field would intuitively put in the same cluster — but the value of the graph is in the edges that aren't obvious: a Research shop and a GovTech vendor working on the same problem statement, or a Fellowship program and a Capacity Building org sharing the same set of funders.

It does not show documented relationships. The data does not encode partnerships, joint programs, alumni flows, board interlocks, or co-investments. An edge in this graph is an inference about likely affinity from public-facing descriptions and tags, not an assertion that two orgs are working together.

Color palette

Primary segments are color-coded consistently across the directory's segment tag column and the network's nodes. The mapping:

  • Research
  • Government
  • Philanthropy
  • Fellowships
  • Community
  • GovTech
  • Advocacy
  • Digital Services & Consulting
  • Investor
  • Capacity Building
  • Ecosystems

Credits

The State Capacity Ecosystem Database is built and curated by the State Capacity Ecosystem (SCE) team. The visualization and interaction layer is maintained by Tal Roded.

Stay in the loop on state capacity events and tools.
Subscribe to our Substack ↗ Explore the Hub →