Complete Trunk Flaky Tests Review 2025

Trunk Flaky Tests Review: Finally, Someone's Tackling the Most Annoying Part of CI/CD

I've been poking around with Trunk Flaky Tests recently, and honestly? I was pretty skeptical at first. Another tool promising to fix flaky tests? Yeah, sure.

Look, if you've worked in any development team for more than like... a week, you know the pain. Tests that pass. Then fail. Then pass again. For no apparent reason. It's maddening. You end up re-running CI pipelines constantly, or worse – your team just starts ignoring test failures because "oh, that one's just flaky." And then real bugs slip through because nobody trusts the test suite anymore.

But here's the thing – after actually using Trunk Flaky Tests for a bit, I get what they're trying to do. Not gonna lie, it doesn't solve everything perfectly, but it does tackle a real problem that most teams just... live with? Which is kinda sad when you think about it.

What is Trunk Flaky Tests?

Trunk Flaky Tests is basically a tool that detects, tracks, and helps you manage flaky tests in your CI/CD pipeline. The whole pitch is that instead of manually hunting down which tests are unreliable, it automatically identifies patterns and gives you data on what's actually flaky versus what's a real failure.

The main hook is automated detection using statistical analysis. It watches your test runs, figures out which ones fail inconsistently, and quarantines them so they don't block your entire pipeline. Honestly? The quarantine feature is pretty clever. Your flaky tests still run, but they don't fail your builds while you're fixing them.

My Real Experience

Alright, let's get into the actual testing. When I first tried Trunk Flaky Tests, my impression was... confusion. The setup wasn't terrible but the docs assumed I knew more about their ecosystem than I did. I had to dig around a bit to figure out how it integrates with our existing GitHub Actions setup.

But once I got it working? Pretty solid. I tested it with our main application repo that had – and I'm not exaggerating – at least 15-20 tests that would randomly fail. The kind where you'd just hit "re-run" and they'd pass. Super annoying.

Within the first few runs, it started flagging tests as potentially flaky. The detection isn't instant though. It needs multiple runs to build up enough data to be confident. Which makes sense, but if you're impatient (I am), it feels slow at first.

What surprised me was how it presents the data. You get this dashboard showing failure rates, which specific tests are problematic, and even patterns about WHEN they fail. Like, some of our tests were more likely to fail during peak hours when our test database was under load. That's... actually useful information? Not something I would've figured out manually without spending hours digging through logs.

The quarantine feature is where things got interesting. Once a test is marked as flaky, it still runs but doesn't block merges. This is honestly kind of controversial on my team. Some people love it because we're not stuck waiting. Others hate it because they think we're just ignoring problems. To be fair, both perspectives are valid.

I will say – the integration with our pull request workflow is smoother than I expected. Flaky test results show up clearly labeled, so you know what's what. It's not hiding information, just categorizing it differently.

Key Features

Automatic Flaky Test Detection

This is probably the core value prop. The system watches your test runs and uses statistical analysis to determine what's actually flaky.

How it works: it's tracking pass/fail patterns across multiple runs. If a test fails 30% of the time without code changes, that's flaky. If it suddenly starts failing consistently after a commit, that's probably a real bug.

Honestly? It works better than I thought. But it's not magic. You need enough test runs for the data to be meaningful. If you only run tests a few times a day, it'll take longer to identify patterns.

Test Quarantine

When a test is identified as flaky, you can quarantine it. This means it still runs and reports results, but it won't fail your CI pipeline.

I had mixed feelings about this at first. Isn't this just... ignoring the problem? But here's the reality – flaky tests ALREADY get ignored by teams. People just re-run pipelines or merge anyway. At least with quarantine, there's visibility and tracking. You're acknowledging the issue instead of pretending it doesn't exist.

The quarantine isn't permanent either. You're supposed to fix the test, and once it's stable again, it comes out of quarantine. In practice though... yeah, some tests have been in quarantine longer than I'd like to admit.

Failure Pattern Analysis

This feature breaks down WHY tests might be flaky. Time-based patterns, infrastructure issues, test order dependencies, that kind of thing.

Not gonna lie, this is where Trunk Flaky Tests really shines. One of our tests was failing specifically when it ran after another specific test. Test order dependency that nobody had caught. The pattern analysis surfaced it pretty clearly.

It's not always that obvious though. Sometimes the patterns are like "fails more often on Tuesdays" and you're left thinking... okay, what do I do with that information?

CI/CD Integration

Works with GitHub Actions, GitLab CI, CircleCI, Jenkins – basically everything you'd expect. The setup varies depending on your platform but it's generally pretty straightforward.

The GitHub Actions integration is what I used. You add their action to your workflow file, and it starts collecting data. That's it. No complicated configuration or infrastructure changes needed.

One minor annoyance: it adds a bit of time to your CI runs. Not a ton, but noticeable. Maybe an extra 30 seconds to a minute? For larger test suites it might be more.

Historical Tracking

You get a history of test reliability over time. This is actually super helpful for understanding if you're making progress on flaky tests or if things are getting worse.

We could see that after a big refactor, our flaky test count went DOWN. That felt good. Then after another sprint, it went back up. That felt... less good. But at least we had visibility.

Team Notifications

You can set up alerts when new flaky tests are detected or when quarantined tests need attention. Integration with Slack and other tools.

I haven't used this extensively yet because honestly, we get enough notifications already. But I can see it being useful for larger teams where ownership of tests is more distributed.

Pricing

Here's where I get annoyed. The pricing isn't super clear on their website. Classic enterprise software move.

Based on what I could find, Trunk Flaky Tests is part of Trunk's broader platform. There's a free tier for smaller teams, but the limits are... fuzzy. Something about number of monthly test runs or users, but the exact numbers aren't prominently displayed.

For creators like me – well, I'm on a team, so I'm not paying personally. But if you're a small startup or open source project, the pricing ambiguity is frustrating. You have to contact sales to get real numbers, which always feels like they're going to quote you based on how desperate you seem.

There's supposedly a free option for open source projects. That's cool, I guess. But again, not a lot of detail about what qualifies or what the limits are.

Check out Trunk Flaky Tests if you want to see their current pricing – maybe they've made it clearer since I looked. But be prepared to potentially talk to sales if you need real numbers.

Pros

• The automatic detection actually works. Like, it finds flaky tests that you KNOW are flaky but also surfaces ones you didn't realize were problematic.
• Quarantine feature is controversial but practical. Better than just ignoring failures.
• Pattern analysis can surface root causes you wouldn't find manually. The test order dependency detection alone saved us hours.
• Setup is relatively painless. If you're already using modern CI/CD, integration is straightforward.
• Dashboard is clean and actually useful. Not just data vomit – it highlights what matters.
• Historical tracking helps you measure progress. Nice for team retrospectives when you can say "we reduced flaky tests by 40%."
• Doesn't require changing your test code. It's analysis layer on top of your existing tests.
• Works across different testing frameworks. Doesn't matter if you're using Jest, pytest, JUnit, whatever.

Cons

• Pricing transparency is terrible. Why do I have to talk to sales to get basic pricing info?
• Takes time to build up useful data. If you want instant results, you'll be disappointed.
• Adds overhead to CI runs. Not huge, but noticeable on large test suites.
• The quarantine feature can become a dumping ground. "Oh it's flaky, just quarantine it" becomes the default instead of actually fixing tests.
• Documentation assumes familiarity with their ecosystem. If you're just using Flaky Tests standalone, some docs don't apply and it's confusing.
• No mobile app or notifications outside of what they integrate with. Everything is web dashboard.
• The statistical analysis is a black box. You can't really tweak the sensitivity or thresholds for what counts as flaky.
• Limited in what it can tell you about HOW to fix flaky tests. It identifies them, but you're still on your own for the actual fix.

Who Should Use It?

Honestly? This is best for development teams that are already feeling the pain of flaky tests. If you're constantly re-running CI pipelines, or your team has started ignoring test failures, Trunk Flaky Tests could save you a lot of frustration.

It's especially good for teams with:
- Large test suites (hundreds or thousands of tests)
- Multiple contributors where test ownership is distributed
- Tests that interact with external services or databases
- Fast-moving codebases where test maintenance is hard to keep up with

Who shouldn't use it? If you're a solo developer with like 20 tests, this is overkill. Just fix your tests manually. If you're working on a project where tests run infrequently (maybe embedded systems or something), there won't be enough data for the detection to work well.

If you're a perfectionist who wants every single test to be rock solid before quarantining anything, you'll probably be disappointed. At that point, you might as well just invest the time in fixing tests manually and implementing better test isolation practices from the start.

Alternatives

The closest competitors are probably BuildPulse and Launchable. Both tackle similar problems around test flakiness and CI optimization.

BuildPulse focuses more on the analytics side – giving you deep insights into test performance and reliability. From what I've seen, their reporting is more detailed but the setup is more involved.

Launchable does predictive test selection – running the tests most likely to catch bugs based on what code changed. Different approach to the same underlying problem of CI taking too long and tests being unreliable.

There's also the DIY approach – writing scripts to track test results over time and identify patterns yourself. If you've got the engineering time and want full control, that's an option. But honestly, most teams don't have the bandwidth.

Google's approach with their internal "Test Assured" system is interesting but not available externally. Some of the concepts have influenced these commercial tools though.

Final Verdict

Look, I'm not saying Trunk Flaky Tests will change your life, but it has its place. If you're drowning in unreliable tests and your team is losing trust in the CI pipeline, it's worth trying.

The automatic detection is legitimately helpful, but the quarantine feature is a double-edged sword. It solves the immediate problem of flaky tests blocking progress, but it can also enable teams to procrastinate on actually fixing the root causes.

I'll probably keep using it because the visibility alone is valuable, even though the pricing opacity annoys me. Sometimes "good enough and fast" beats "perfect and time-consuming." And honestly, having data on which tests are problematic is better than the previous system of "Dave thinks this test is flaky but Sarah swears it's fine."

The pattern analysis has caught issues we wouldn't have found otherwise. That's real value. But it's not a magic solution – you still have to put in the work to fix your tests.

Rating: 3.5/5 stars

It's a solid tool for a real problem, but the pricing confusion and the risk of quarantine becoming permanent limbo for unfixed tests hold it back from being great.

Bottom line: If you've got a CI/CD pipeline that's constantly blocked by flaky tests and don't mind the somewhat unclear pricing model, Trunk Flaky Tests is worth checking out. Just be prepared for it to take a few weeks to gather enough data to be truly useful, and make sure your team commits to actually fixing quarantined tests instead of letting them rot.

To be fair, most DevOps & CI/CD Testing tools are still evolving. The whole space is relatively new in terms of dedicated tooling. But for what it does – identifying and managing flaky tests so they don't block your entire workflow – it gets the job done. Just don't expect miracles. You still need to fix your tests eventually. This just gives you breathing room and data to do it effectively.