• 9point6@lemmy.world
    link
    fedilink
    arrow-up
    12
    arrow-down
    4
    ·
    edit-2
    5 months ago

    Good tracing & monitoring means you should basically never need to look at logs.

    Pipe them all into a dumb S3 bucket with less than a week retention and grep away for that one time out of 1000 when you didn’t put enough info on the trace or fire enough metrics. Remove redundant logs that are covered by traces and metrics to keep costs down (or at least drop them to debug log level and only store info & up if they’re helpful during local dev).

      • 9point6@lemmy.world
        link
        fedilink
        arrow-up
        5
        ·
        5 months ago

        Well I didn’t say anything about perfectly clean, but I agree, it’s very nice to work on my current projects which we’ve set up our observability to modern standards when compared to any of the log vomiting services I’ve worked on in the past.

        Obviously easier to start with everything set up nicely in a Greenfield project, but don’t let perfect be the enemy of good—iterative improvements on badly designed observability nearly always pays off.