• 0 Posts
  • 6 Comments
Joined 5 months ago
cake
Cake day: June 11th, 2024

help-circle

  • My profesional experience is in systems administration, cloud architecture, and automation, with considerations for corporate disaster recovery and regular 3rd party audits.

    The short answer to all of your questions boil down to two things;

    1: If you’re going to maintain a system, write a script to build it, then use the script (I’ll expand this below).

    2: Expect a catastrophic failure. Total loss, server gone. As such; backup all unique or user-generated data regularly, and practice restoring it.

    Okay back to #1; I prefer shell scripts (pick your favorite shell, doesn’t matter which), because there are basically zero requirements. Your system will have your preferred shell installed within minutes of existing, there is no possibility that it won’t. But why shell? Because then you don’t need docker, or python, or a specific version of a specifc module/plugin/library/etc.

    So okay, we’re gonna write a script. “I should install by hand as I’m taking down notes” right? Hell, “I can write the script as I’m manually installing”, “why can’t that be my notes?”. All totally valid, I do that too. But don’t use the manually installed one and call it done. Set the server on fire, make a new one, run the script. If everything works, you didn’t forget that “oh right, this thing real quick” requirement. You know your script will bring you from blank OS to working server.

    Once you have those, the worst case scenario is “shit, it’s gone… build new server, run script, restore backup”. The penalty for critical loss of infrastructure is some downtime. If you want to avoid that, see if you can install the app on two servers, the DB on another two (with replication), and set up a cluster. Worst case (say the whole region is deleted) is the same; make new server, run script, restore backups.

    If you really want to get into docker or etc after that, there’s no blocker. You know how the build the system “bare metal”, all that’s left is describing it to docker. Or cloudformation, terraform, etc, etc, etc. I highly recommend doing it with shell first, because A: You learn a lot about the system and B: you’re ready to troubleshoot it (if you want to figure out why it failed and try to mitigate it before it happens again, rather than just hitting “reset” every time).





  • This is fucked.

    I worked in call centers for many years (technical support and sales). I need to hear the customer’s tone; ecstatic, livid, and everything in between. I sit on the other end, shut my mouth, and listen to the whole rant, then calmly offer suggestions. Do they scream some more? Maybe. Do I need to take it personally? Of course not.

    It drives me fucking crazy when some dipshit customer service rep hears one swear word (not even directed at them, like “I hate this fuckin’ thing”, not “you’re a fuckin’ dumbass”) and start in on the “if you keep swearing at me, I’ll end the call”. Grow up, you work in a service industry, and your company probably fucked up.

    My favorite calls were the ones where someone called to cancel and tore up their voice yelling about all the reasons our product was gabrage. Very, very roughly, about 15% of the time there was nothing I could do (even if I fixed the problem, they have lost faith and will get their money back, or sue trying, so I just refund and move on). Another 25% was me fixing the problem and offering a credit because we fucked up. About half the time, its something stupid and simple and they get their problem solved, and the rest of the time was some absolutely crazy broken shit that makes me work with someone two tiers above me for a few hours fixing it (for everyone, not just that caller), then the customer is so happy they renew everything for a year because they know they’re gonna get great support.

    I loved those calls. They were the reason I kept showing up to work. I learned a ton in those jobs, and my favorite thing was hearing someone go from completely apoplectic to surprised and elated that everything was fixed.