Methodology
Last updated: May 25, 2026
CompareDev is a data tracker for developer SaaS. Every dashboard wraps automated public-data feeds with a short editorial verdict. This page documents exactly how the data is collected, scored, and updated — so you can audit any number on the site.
What CompareDev does and does not do
Does:
- Tracks pricing changes over time, sourced from the Wayback Machine and re-checked on a cron.
- Pulls 90-day uptime from each vendor’s public status-page JSON, refreshed daily.
- Aggregates sentiment from G2, Capterra, Reddit, and Hacker News, with every quoted line linked back to its source.
- Tracks changelog cadence from public RSS feeds and GitHub release feeds.
- Publishes a composite CompareDev Score using the rubric below.
- Writes a short editorial verdict per tool — best for whom, avoid if what.
Does not:
- Personally deploy these tools in production. CompareDev is run by one engineer; honest first-hand testing of 200 tools is not possible and pretending otherwise is dishonest.
- Accept paid placements, sponsored posts, or review-for-review trades.
- Fabricate ratings, benchmarks, or quotes. If we don’t have a citation, we don’t print it.
If first-hand testing matters for your decision, treat each dashboard as a data brief and supplement with the vendor’s free trial.
Data pipelines
Pricing history
- Source: archive.org/wayback snapshots of each vendor’s public pricing page.
- Cadence: historical pull on dashboard creation; weekly re-check via cron after launch.
- What we record: plan name, monthly price (annual equivalent if only annual is shown), seat tier, free-trial terms, and the snapshot date.
- What we do not record: custom enterprise quotes (these are not publicly listed). Where a plan is “contact sales,” the row reads Contact sales and is not used in the Score.
Uptime (90 days)
- Source: public status-page JSON endpoints (Statuspage.io, BetterStack, OpenStatus, Instatus, or the vendor’s own).
- Cadence: daily snapshot.
- What we record: count of incident days in the rolling 90-day window, weighted by severity (
major× 1.0,minor× 0.5,maintenance× 0). - Caveats: we report what the vendor reports. Vendors who do not run a public status page get a No public status page badge and are not eligible for the uptime portion of the Score.
Sentiment aggregation
- Sources: G2, Capterra, Reddit (
r/devops,r/programming, tool-specific subs), Hacker News (Algolia search API), GitHub Issues (when the vendor’s product is open source or has a public issue tracker). - Cadence: monthly re-sweep, plus event-driven re-sweeps when a major release or pricing change is detected.
- What we record: the 3–5 most-mentioned positives and negatives over the last 12 months, each linked to its source. Volume thresholds: at least 8 distinct posts/reviews must mention a theme before it appears in Pros/Cons.
- What we do not record: anonymous Reddit comments below score 5, deleted reviews, or paid placements on review sites flagged by their own disclosure.
Changelog cadence
- Source: vendor’s public RSS / Atom feed, GitHub releases, or changelog page parsed monthly.
- What we record: count of releases in the last 90 days and the date of the most recent.
The CompareDev Score
A composite metric, 0–100, computed from public signals only. The rubric is fixed; we do not tune it per tool.
| Component | Weight | How it’s calculated |
|---|---|---|
| Pricing transparency | 20 | Full marks if all plans are publicly priced; partial if some require sales contact; zero if all are “contact sales.” |
| Uptime (90d) | 25 | 100 minus severity-weighted incident days × 2, floored at 0. |
| Sentiment balance | 25 | Ratio of net-positive themes to total themes in our sentiment aggregation. |
| Changelog cadence | 15 | Logarithmic scale on releases per 90 days; full marks at ≥12. |
| Documentation depth | 15 | Manual rubric (presence of API docs, quickstart, working examples, public roadmap). Re-rated quarterly. |
The Score is shown on each dashboard with a “How this was computed” link back to this page and a breakdown of the inputs.
What we don’t score
- Tools with fewer than 10 public reviews across all sentiment sources are shown as “Insufficient public data” rather than scored.
- Tools younger than 6 months are tracked but show a “New — score pending more data” badge.
Update cadence
- Pricing rows: weekly cron, plus event-driven re-check when sentiment sources flag a pricing change.
- Uptime: daily.
- Sentiment: monthly sweep.
- Changelog: monthly sweep.
- Editorial verdict: revised in place whenever any signal moves materially. Each dashboard carries a per-row timestamp, not a single page-level date, so you can see exactly when each number was last refreshed.
Corrections
If a number on this site is wrong, file it at /corrections or email via the contact page. Every correction is logged publicly with the date, the change, and the source that prompted it.
Conflicts of interest
Affiliate revenue does not affect the Score, the Pros/Cons, or the editorial verdict. See /disclosure for the full policy.