DevSal
Data pipeline · REST API · React dashboard

Developer compensation intelligence, end to end

DevSal ingests raw salary reports from public sources, normalizes messy job titles and locations into a canonical schema, runs a statistical engine over each cohort, and serves the results through a REST API and a dark-mode analytics dashboard.

How it works

The pipeline

Five stages, from raw CSV rows to percentile-ranked cohorts. The statistical layer is pure functions with no database dependency, so the same code powers ingestion, the API, and ad-hoc analysis.

01

Ingest

Stream CSV rows from public sources (Stack Overflow, H1B, levels.fyi, Glassdoor) with a configurable column map. Invalid or non-positive salaries are rejected and counted.

→
02

Normalize

Regex patterns map ~free-text titles to 16 canonical roles & 8 seniority levels; locations parse into city / region / country; currency converts to USD via an FX table.

→
03

Model

PostgreSQL schema with typed enums, foreign keys to roles / locations / data_sources, a salary_records fact table and a precomputed salary_stats rollup.

→
04

Analyze

Statistical engine: interpolated percentiles, IQR-fence outlier removal, cost-of-living adjustment, confidence × source-reliability weighting, histograms, skewness.

→
05

Serve

Express REST API (10 endpoints, Helmet + rate-limit + compression) feeding a React 18 / Recharts dashboard with Overview, Compare, Percentile and Explorer views.

About the data on this page. The repository ships a seed generator instead of a bundled dataset. Every figure below is the real output of the pipeline's statistical engine run over a fixed 2,500-record synthetic sample (generateDataset(2500), seeded for reproducibility). The distributions are modelled from role × location × seniority × company multipliers with Gaussian noise — realistic in shape, but not collected market data.

What the engine found

Global picture — 2,500 records

Base salary distribution
USD base salary, $25k buckets — outliers retained for the histogram
Descriptive statistics
Base & total comp, IQR-fence outliers removed

Both distributions are right-skewed (positive skew): a dense band of mid-market salaries with a long thin tail of staff-plus and big-tech packages pulling the mean above the median.

Seniority is the dominant signal

Median base salary by experience level

The clearest driver in the dataset. Median base compensation grows roughly 7× from intern to fellow, with the steepest jumps at the senior and distinguished thresholds.

Remote vs. on-site
Base salary, same cohort definitions

Experience mix of the sample
Share of records at each level

Role premiums are real but narrower

Median base salary by role

Across all seniority levels combined, the spread from the top role to the bottom is about $66k of median base — smaller than a single step up the seniority ladder. AI/ML, security and management lead; QA and design trail.

Geography: raw vs. cost-of-living adjusted

Where the money goes furthest

San Francisco and New York top raw median base salary. But once each city's median is divided by its cost-of-living index (NYC = 100 baseline), the ranking inverts — lower-cost US hubs like Austin, Atlanta and Denver deliver the most real purchasing power, while the Bay Area premium mostly cancels out.

Top locations — raw median vs. COL-adjusted median (USD)
Emerging-market locations (Bangalore, Lagos, São Paulo) sit far below on raw salary, but their low COL indices push their adjusted figures back into the same band as mid-tier US cities — a reminder that headline salary comparisons across borders are close to meaningless without normalization.

Provenance

Data sources & reliability weighting

Every record carries a source and a per-source reliability score (0–1). The engine's weighted-mean function multiplies each record's own confidence by its source reliability, so a levels.fyi or H1B data point counts for more than a scraped or survey one.

Implementation

Stack & running it

Backend
  • Node.js + Express 4 REST API (ES modules)
  • PostgreSQL via pg, typed enum schema + migration runner
  • csv-parse streaming ingest with pluggable column maps
  • Helmet, CORS, express-rate-limit, gzip compression
  • Zero-dependency statistical engine (utils/statistics.js)
Frontend
  • React 18 + Vite 5, React Router, SWR data fetching
  • Recharts visualizations, Zustand filter store
  • Four views: Overview, Compare, Percentile, Explorer
  • Dark design-token system, JetBrains Mono / Outfit
# backend
cd backend && npm install
node src/migrations/run.js      # create schema (needs DATABASE_URL)
npm start                       # API on :3001  — serves the in-memory
                                # 2,500-record dataset if no DB is seeded

# frontend
cd frontend && npm install
npm run dev                     # dashboard on :3000

# ad-hoc ingest
node src/services/ingest.js --file=data.csv --source=levels.fyi

API surface: /api/overview, /api/salaries, /api/salaries/stats, /api/salaries/percentile, /api/salaries/compare, /api/salaries/distribution, /api/roles, /api/locations, /api/sources, /api/health. All read-only, all filterable by role, category, location, experience, company, company size, remote and salary range.