TL;DR possible thumbnail and local image outage tonight

Tonight I plan to update the image engine behind lemmy, called pictrs, to the latest stable version and to also perform a migration to a different database (sled to postgres).

In my testing there was minimal outage, but it did take quite some hours to do the migration (12+, though my test server is a lot slower than the production one).

Once the migration is complete, I intend to turn on a thumbnail cache cleaner that will significantly reduce the size of the thumbnail cache on the server. Lemmy currently saves many thumbnails locally, and never deletes them. This means we have over 260GB of images that get backed up each night for no reason. Lemmy makes it hard to tell the difference between cached images and user uploaded ones, but the cleaning tool will go through each post in the database, find the thumbnail associated with it, and delete it if it’s more than a month old.

Any questions, let me know!

Also, we are still having issues with lemmy.world. They are testing out new tools to block kbin traffic if they start to see huge amounts of activity coming from them, which will hopefully mitigate this to some extent. There is a tool running to help get posts and comments to us in real time, but it won’t work for votes which means you’ll see lots of lemmy.world posts and comments with zero votes, and the votes will be delayed in arriving by up to 1.5 days.

Feel free to ask questions about that too!

    • DaveOPMA
      link
      fedilink
      arrow-up
      2
      ·
      7 months ago

      Morning update - migration is still running. I’m still not exactly sure what the impact is. Many thumbnails seem to come via other instances, and the lemmy.nz ones I can find seem to work but are probably cached by cloudflare. In any case, it should be minimal impact.

      • DaveOPMA
        link
        fedilink
        arrow-up
        1
        ·
        7 months ago

        Afternoon update - migration is still running…

        • DaveOPMA
          link
          fedilink
          arrow-up
          1
          ·
          7 months ago

          Night update - this is definitely taking longer than I expected…

          • DaveOPMA
            link
            fedilink
            arrow-up
            1
            ·
            7 months ago

            Next morning update - still going 😑

            • DaveOPMA
              link
              fedilink
              arrow-up
              1
              ·
              7 months ago

              Still going 🤯 - but have sought help and have made an adjustment. Hopefully that solves it!

              • DaveOPMA
                link
                fedilink
                arrow-up
                2
                ·
                7 months ago

                It’s done!

                I’ll do the cache cleaner probably tomorrow, but this shouldn’t affect uptime so will do it whenever I get a chance.