End to end and smoke tests give a really valuable angle on what the app is doing and can warn you about failures before they happen. However, because they’re working with a live app and a live database over a live network, they can introduce a lot of flakiness. Beyond just changes to the app, different data in the environment or other issues can cause a smoke test failure.

How do you handle the inherent flakiness of testing against a live app?

When do you run smokes? On every phoenix branch? Pre-prod? Prod only?

Who fixes the issues that the smokes find?

  • AggressivelyPassive@feddit.de
    link
    fedilink
    arrow-up
    5
    ·
    1 year ago

    Shouldn’t you find out, why the smoke tests keep failing? It’s not a good sign, if you can’t even guarantee stability in a controlled environment.