Video Case Study #2: TopClassActions.com Sean TierneyMarch 27, 2019April 2, 2019 TopClassActions.com is a consumer advocacy site offering consumers the latest information on class action settlements. With traffic in excess of 1.25MM visitors per month and a weekly email blast that goes out to 700k+ subscribers, TCA plays a high-stakes game of maintaining bullet-proof uptime during these weekly email blasts. Every minute of downtime during one of these high-traffic periods represents thousands of dollars in lost revenue and irreparable loss of potential clients for the advertising attorneys. On top of the monetary stakes involved in maintaining flawless uptime through traffic spikes, TCA faces a unique security challenge. Recruiting the masses to become plaintiffs against powerful corporate interests significantly compounds the hosting challenges and paints a big bullseye on TCA for hacking and DDoS attempts.TCA came to Pagely seeking help with these security and scaling challenges. What follows is a one-hour conversation in which we deconstruct each of the various challenges and explain the solutions Pagely implemented to help TCA navigate them successfully. TCA has quintupled its business since being a Pagely client and now ranks #1 for “class actions settlements.” Enjoy this enlightening conversation and feel free to ask questions via comment box below. And if you’re suffering from similar scaling or security challenges get in touch with us and let’s get you sorted out the way we did TCA. Show NotesTime Topic0:00:00 Teaser0:02:08 Welcome and backstory on TCA0:09:31 Challenge #1: Gravity Forms plugin viewcount issue that only manifests at scale0:10:34 How did we patch the plugin in a durable way so the fixes would survive plugin updates?0:12:25 Challenge #2: Poor cache hit rate0:13:11 How the PressCache plugin works0:17:53 Challenge #3: Getting killed with bot traffic0:20:35 “We used to block everyone but the US from accessing us because we had so many hacking attempts”0:21:34 Robots.txt as the first line of defense. Rate-limiting by IP and user agent as the 2nd line of defense0:23:47 Challenge #3: Brute force login attempts0:26:50 “Security is criticlal. I get one to two cease & desists per month from billion-dollar companies”0:28:24 Challenge #4: Easy Digital Downloads session cookie busting cache0:31:23 Challenge #5: external Cloudflare service misconfig creating an infinite redirect loop0:35:27 “One of the huge benefits to our new ARES gateway is Let’s Encrypt is built right in”0:37:35 Challenge #6: excessive uncached admin AJAX calls + image URL’s with spaces0:42:10 How much does downtime cost you during one of these weekly newsletter sends?0:48:14 Review the highlights of their PressFormance audit reportLinks mentioned in the interviewTop Class ActionsJoomlaPressCacheCloudflareGravity FormsGearman/WP MinionsLet’s EncryptAdmin AJAX Pagely’s new ARES GatewayTranscriptScott Hardy: 00:00:00 I did what anybody would do probably my position which was throw money at the hardware. Let’s max out the platform we’re on and we do that and it would work for a couple of months, but then our subscribers kept growing and eventually we’d start crashing the platform again. So if they try to submit it to us and it does not work, then they’re gone forever and our client could miss out on us six or seven figure case and then we lose upwards of anywhere from eight to $10,000 a month in revenue from just one case. Once we hit about 250,000 subscribers and we started sending out that big blast of emails, we started crashing. It took your guys about 10 minutes to say “oh yea you’ve got all these active forms on your website. There’s this, it’s five lines of code and if we comment out those lines of code, we’re going to take the pressure off the database. You won’t crash anymore.” And you did that and we didn’t crash.Arman Zakaryan: 00:01:02 This is, you know, this is one of those situations where it’s good to have a managed WordPress company hosting your site. Because they have engineers that will, we’ll be able to understand what’s going on. And secondarily like the ability to engage and make changes on your behalf. Arman Zakaryan: 00:01:23 You just can’t bring one expert into a business and have them even if they are extremely experienced, have them have the breadth of knowledge that you need. When you guys deal with hundreds of different companies on a daily basis. It’s just don’t necessarily go out hire to super techiest most expert guy you can get look at your money and see if you can bring that in to a partner that will, that will handle it so you can focus those big dollars on something else to help you actually grow your business and not just support it.Sean Tierney: 00:02:09 Okay, we will get started then. This is Sean welcome people who are tuning in. This is our second video case study. And on the line with us is Scott Hardy. Of Top class action. Um, Scott, I’ll let you kind of introduce what that business is. We’ve been long-time acquaintance and friends and uh, maybe you can kind of explain.Scott Hardy: 00:02:32 sure. Yeah, I’m Scott Hardy, a cop crash actions to sort started top class actions 11 years ago. And it was all with the goal of bringing consumers the unclaimed cash out there from class action settlements. And so I had the idea when I was slipping through the Internet and I was, I saw these class action settlements and I, I said, man, there’s gotta be a website which tracks he’s easy to claim settlements. Well there wasn’t. So I built it and now we’ve, we’ve grown leaps and bounds over the years. Now we get 1.2 5 million viewers a month. And we originally started on a Joomla platform because originally it was supposed to be me and then my it engineer guy and my kind of executive senior executive money raiser guy. And then when it came that time to actually put in money into the business, those two guys dropped out.Scott Hardy: 00:03:28 Well, even just me as I used to be an it manager and into it. So I’ve got to, I’ve got just enough knowledge to get myself in trouble and to think I know what the heck I’m talking about. But, uh, and, and really to infuriate engineers because now I’m at executive that says, hey guys, if we just can we build this database? You can just spin that up in what a couple of days, right? That’s like a day job. Like, oh Scott, that’s six months and you know, hundreds if not thousands of hours of engineering time. But so we put it all on all up on Joomla. And it was just me, you know, writing articles, adding plugins and you know, making the website grow. And that was okay when we had tens of thousands of viewers, but we kept maxing out web platforms. And then Juma at that point, it wasn’t very efficient.Scott Hardy: 00:04:18 So after looking different platforms, we said, okay, let’s go to wordpress. And so we did, we migrated over to wordpress and our business is largely built on user submissions. So you know, we get paid, we don’t charge consumers to access a website. We get paid by attorneys to advertise on the website. And what that entails is we run the advertise on the website. People go, yes, that happened to me. Um, I’m going to go ahead and type in my information and submit the information directly to the attorneys. So we’ve got a whole bunch of submission forms right now. I just looked and we’ve got over a thousand plus submission forms from everything that we’ve done the past. But actively we’ve got anywhere from 50 to 75 campaigns going at a time. And it turned out as our traffic kept growing, you know, we also have a big email list and our email list now goes out to 700,000 subscribers and that goes out every Tuesday.Scott Hardy: 00:05:23 And so once we hit about 250,000 subscribers and we started sending out that big blast of email, we started crashing. And you know, I’m not a coder and I don’t have a full time engineer at that point. This was still a side Gig for me that was just bringing in good money and paying for vacations, but you know, you gotta be up. Right. And so I did what anybody would do, probably my position, which was throw money at the hardware. All right, well let’s Max out the platform we’re on. And we do that and it would for a couple of months, but then our subscribers kept growing and eventually we’d start crashing the platform again. Once that newsletter we go out. And so we bounced among a number of different providers and at that point, you know, we’ve got to a point where I was doing the job full time and I would flip out and you’ve, you’ve had to deal with me when I flipped out an occasion shot and I’m like, I’m like, oh my gosh, what the heck?Scott Hardy: 00:06:21 We’re crashing. You know, I’ve got thousands and thousands of people hitting the website and they can’t get to us. I’m losing money. The Sky is falling. I feel like I’m dying here. What are we going to do? And different, different managed wordpress platform. It’s giving different answers. Some of them had said, oh, it’s your code. And I’m like, Hey, I get it. We went from Joomla to wordpress. Our code sucks. Our backend code sucks. I get it, but we still need to be up. So some said, well, pay these guys. Let’s pay these guys $10,000 you’re going to optimize your code. So you write a check for 10 grand. And then Lo and behold, a week later, the website crashes again. Well crap that didn’t work. Go to it. And I go to different provider with bigger, more robust servers. Scott, we won’t crash. Great. We, we go ahead and send out the newsletter.Scott Hardy: 00:07:12 Boom, we crash crap. And they say, well Scott, pay these guys $35,000 and they will optimize your code and then you won’t crash man, that we just did this kind of code scrubbing. I know our code is great, but we don’t have a lot of video. You know, our stuff is at bandwidth heavy. It’s all text. Sorry, my dog is going off. And so, uh, you know, we’re just a text website with some pictures. So come on guys, don’t, it’s not like when I was with limelight networks and we’re still streaming gigabytes worth of video here, that’s not the case. Why are we going to go and crash? And so then we would, um, we, we ended up tied, trout was talking to you, Sean. And I said, hey, I need a new platform that can handle these huge bursts of traffic. And you said, Scott, I’m working with Paisley now.Scott Hardy: 00:08:10 We can go ahead and get this working for you. And we went to Pagely and you said, great. You know, hey, we, you know, we’re built an Amazon, we can go ahead and structure this however we need to. We can scale for you. And we went to you guys and we crashed. And I went, ah, what’s going on? And then talking to your engineering team, because of course I don’t have that back engineering team. I’ve got web developers, but not people that know the hardware. It took your guys about 10 minutes ago. Oh yeah. Your forms and you’ve got all these active forms on your website. There’s this, it’s five lines of code. And if we comment out those lines of code, we’re going to take the pressure off the database. You won’t crash anymore. And you did thatSean Tierney: 00:08:54 and we didn’t crash.Sean Tierney: 00:08:56 Yeah. Let me actually, this is a good segway because I’ve got a list of issues here. Um, I also just wanted to quickly introduce for the folks listening, our minds as a Korean is our head of hosting off and he’s on the call as well. So, and this is the first time you guys have met. Again, this is all, what’s interesting about these video case studies is you know, our mind only through support history and tickets and chats. And so this is actually the first time that we’re all the three of us talking face to face.Arman Zakaryan: 00:09:27 Yeah.Sean Tierney: 00:09:30 Cool. So Scott, what I did is I actually went back and read through half of your support history. There was like 140 tickets and I read about 70 of those chronologically started from the beginning when you guys became a customer. Um, and the issue that you’re referring to, like right, right away, I notice like within, I think a week of you guys joining Pagely, um, Oscar discovered that the gravity forms plug in because you guys get so much traffic. This is a plugin, which normally doesn’t have this issue for most people, but because you guys get so much that you can get to have so many pages and so many instances of these forms and so many visitors that it logs, it does it a database query on every page view with the view count and that that was the problematic thing. And so Oscar, it I believe, was able to determine, uh, you know, if we just comment out and patch the plugin, we can remove that issue and then it’s not a problem.Sean Tierney: 00:10:23 So I think that was the first, or at least the one that I have here pulled out that was like the first real thing that we solved. It seemed like, mmm. And Armand, correct me if I’m wrong on this. And so did we, I know at least initially it was done in such a way where every time gravity forms issued an update, it was stomping on Oscar’s changes. But I think since then we’ve been able to do it in a more durable way. And I was wondering if you could maybe just comment on what that is. Yes, so that was just using the ad filter functionality.Arman Zakaryan: 00:10:55 Press a to load a separate must use plugin. So that just lays on top of whatever settings there are that there are in the, in the plugin itself and just add something on top of it. Uh, so whenever the plugging gets updated you don’t lose that change cause you’re not editing the plugins code. It’s in a separate path that’s a separate file as then you plug in.Sean Tierney: 00:11:18 Cool. And so this is like for the people listening, I guess the best analogy there is this is the equivalent of when you use like a child theme so that the updates to the parent theme don’t stomp on your updates or changes that you’ve made it. This is essentially the same thing where a patch can be issued to a plug in a and it’s not going to get overwritten when the new plugin updates are coming out. But okay. So Scott, back to your situation. So that sounds like that was really the first thing that we solve for you guys. There was that gravity forums issue. Um, and at the time you said you have 700,000 subscribers. Now I think back in the day I logged it, it looked like 185,000 subscribers at that time. So then I can only imagine, uh, you know, you’ve, you’ve six x that number nearly, which is pretty impressive or about five exit. Um, okay, so I’m gonna, I’m gonna skip ahead here and I have these all large chronologically, but now we’re talking, so you guys became a customer in May of 2015, somewhere around August, late August of 2015. Uh, it sounds like we, we discovered an issue with a poor cache hit rate and our mom, maybe you can kind of just, uh, briefly deconstruct what the nature of that was. What was happening there.Arman Zakaryan: 00:12:39 Yeah. So whenever you use a page caching layer on your wordpress site, you need to either the account for purging to occur whenever you update content. Uh, so that users don’t get outdated pages where you need to be doing a cash control Max stage headers to say when, when a page is due to expire. So one of the benefits of Pagely is we take care of those details for you. You got some patient management plugins that come included with your APP by default. And what are easily, uh, press cash plugging does is it keeps the lookout for certain events happening in wordpress when you’re creating a new post or page, uh, when you’re updating something, when you’re deleting something, those events trigger a purse to happen with our, with our own a caching layer. And so, uh, some of that can happen as, you know, certain plugins will be creating posts when they’re taking other operations.Arman Zakaryan: 00:13:44 Uh, if you know, you have someone signing up for the newsletter, they submit that that’s creating a post and then they’re sending them an email with a link to that post. That’s like a personalized page. Um, so that activity can create a perfect storm if you have a lot of people doing that. And so, uh, one of the things that, uh, we can do that gives us flexibility with our platform is we can exempt certain pages from being affected by those, uh, purge request. So, uh, we can keep the homepage from being purged and that usually keeps things at bay if there’s a lot of people wants to the home page. And then we can also exempt certain post types from issuing a purge. And so that with that second one is really important to have visibility on when that happens. So we do have logging of that one. It does a skip purging for a specific post type of, they know, we can figure a, then it will put that in a locked file so you can just kind of have a sanity check and make sure it’s working as expected. So that those optimizations, uh, kept the system running more stable, less work for the server to go and process all this burgers in the first place and that more more cached at rate. Uh, which results in faster load times for visitors.Sean Tierney: 00:15:05 And so to be clear why this is issue, just real quick, one secondScott Hardy: 00:15:10 question for you. Go ahead. No, go ahead.Sean Tierney: 00:15:14 Oh, I was just going to say real quick. So the reason this is so bad, this isn’t just like someone missing cash on. Like when you have people that come in and they have something like a dynamic or they’re filling out a form, it Mrs. Cash for that one person, but this is actually purging cash for the entire site. So now that every subsequent page request, you have like this massive spike of needing to do a bunch of dynamic processing. So no cash is, it is being able to be used. Say why did that point? Is that correct?Arman Zakaryan: 00:15:44 So when you’re, when you’re updating that a page, uh, it will, it will only first the cash for that age in addition to the base URLs that we, we always first and those being the home page and like beat you or else. So whenever you update one piece of content, it’s not issuing a fall perch but that, but the homepage being purged alone, whenever there was all these other unrelated things being updated, um, like that activity was enough to cause some lung issues. So we just exempted the homepage from being inclusive.Sean Tierney: 00:16:20 Got It. Sorry, go ahead.Scott Hardy: 00:16:22 Did that happen because that happened because we published so much new content because we publish anywhere from 120 to 150 new articles per week that are all news articles on the, you know, class action, settlement lawsuit side of things. Is that why? Because the homepage wasn’t, you know, it was getting updated so frequently during the day.Arman Zakaryan: 00:16:40 So who was that activity? And it was also each, each newsletter sign up, each visitor doing a sign up, uh, was triggering approach too. So it’s kind of both. Yeah. And especially like during newsletter says it’s, if you have a lot of people doing science, which is a good thing, like you have people engaged on your site and you’re for the use case of top class actions, you’re hopefully like helping people with problems that they real, they have in real life. Um, so it’s a good thing that that’s happening. But making sure that the server Vanessa kind of annulled up the right way so it doesn’t hold down.Scott Hardy: 00:17:16 Yeah. Me Cause now we’re getting 4,000 to 10,000 new newsletter signups a week. I’m so glad you fixed it.Arman Zakaryan: 00:17:24 Really happy that we’ve been able to keep you guys running and running faster room argument.Arman Zakaryan: 00:17:29 Okay.Sean Tierney: 00:17:30 Yeah. I mean the hardest thing to do is to build a business that draws that much interest in it. The point where you stand to lose the most is where it sucks. You know, you send out a newsletter to all these people and you have all those eyeballs. And so it’s critical that we keep it online during that point. Um, so the cash is definitely important for that. Um, so the next issue I noticed, so March 21st of 15, it sounds like you guys get hit with a bunch of Bot traffic and Armand, maybe can you just for the people listening who don’t know what Bot traffic is, could you just briefly summarize like what, what is a Bot?Arman Zakaryan: 00:18:10 Yeah. So Bob is just a system out there on the Internet that anyone can run and it uses your website’s sitemap to determine what content to index is about your state. And so, you know, the, the most well known bots out there like Google being, uh, you know, large names like that. Uh, but there’s a lot of other search engine indexes out there. Uh, you know, they, they exist like all over the world. They’re run by different people. And so if you’ve ever heard of robots dot txt, that’s the most general purpose way to give instructions on, on how those bots should to treat your site. Uh, you can say don’t index this path or only only go at this certain rates, uh, to keep the surf from getting overloaded. Uh, so unfortunately some bots out there don’t really hire those rules. And some of them, uh, they just go as fast as they can.Arman Zakaryan: 00:19:16 And you know, generally it’s up to the site owner to make that decision on like, Hey, I want to block this spot. I don’t really have any interest in. And then indexing my side, you know, uh, but obviously you want to Google or bing or the, the other big names to have, uh, have access to index your site. And those bigger names usually will honor those robots are tax rules. Uh, and some of them actually let you specify more specific settings on their sites. Uh, so yeah, it was just a pretty much a, another, uh, another day in the life that basically for us like, okay, well there’s a bot scraping the say, Scott, do you want this? But scraping your site. Now we’ve locked it. Um, so pretty, pretty cut and dry, uh, incident at that time.Sean Tierney: 00:20:07 And for a normal siren, like an average sized site, let’s say like 20 or 30 pages, a bot is not a super big issue. But because Scott, hi Scott, how many pages do you have? Like hundreds of thousands of pages, right?Scott Hardy: 00:20:20 Yeah, a hundred hundreds of thousands with now. Yeah. Crazy.Scott Hardy: 00:20:27 It’s, it’s, it’s, yeah, it’s too much. Especially because now we just target the US and so we’ve been, we used to block all of the countries from accessing us and that was largely because we get so many hacking attempts from every word, from Korea to Ukraine, Russia, you know, Brazil, North Korea. I mean it’s crazy the number of bots that are just hitting the website, just trying to get access and having to, having to block all of that is crazy. You know, we do use cloudflare as well as your, you’re caching system and there’s so much, maybe there are websites now that all they do is just copy other websites to try to get you to go to their website so they can pass issue. And it’s insane. Especially once you’ve got a larger website with a big following like we do, how much of a target it makes you. So, it’s nice that we’ve got you guys in there kind of looking at the back end and you’re taking a loot, seeing who’s knocking on that door and who’s knocking on that door a million times a second to try to get it in.Sean Tierney: 00:21:32 Right. Right. And our, so I just want to be clear here. So it’s like the way to think about this is robots dot text is for those, those bots that are going to respect it. And you know, those are like the instruction set to the ones that actually listened to what you’re asking them to do. And so Google for instance, if you tell them, hey, I want you to crawl me, you know, once a week between these hours, it’ll, it’ll honor that request. But these more nefarious bots are not going to listen to robots at tech. So they’re just going to hammer the site as hard as they can. And so that’s where the second line of defense comes in, which is this idea of rate limiting by IP address. It sounds like we have some other sophistication there are being able to, you know, block also by user Asian, not just IP address. So we have different ways of recognizing bot traffic and making rules for restricting it.Arman Zakaryan: 00:22:21 Yeah. So some of those mechanisms are, are automatic and just a default part of the system. You don’t really have to do anything out of the box to make it work. Um, but we do, we do have the ability to specify a block rules are rate limited roles, uh, for, you know, all, all types of different qualifiers, whether that’s a Ip address or user agent or a combination of things, uh, the type of requests that they’re making or the HTP refer that they’re coming from. Uh, so that flexibility lets us, uh, sort of, you know, block off things that are obviously that like it’s getting, it may, it may be falling into our general rate limiting roles, but maybe a Spec in, in fast enough. Uh, so we can, uh, we can tighten those rules up. We can put custom rules, um, to fit, to fit what’s happening, to get him to say, and that’s again, that’s just like a normal course of business for us. We don’t really have to go out of our way to break how we normally do things to, to apply those customizations.Sean Tierney: 00:23:29 So we have like an umbrella rule set that applies for all customers, but we also have this ability on a per customer basis or a per site basis. So then do some specific more sophisticated things if, if there is unique traffic. That’s what we were trying to exclude for that site. Right. Exactly. Cool. All right, well this is also a good segway because this is related to security and I noticed it looks like in January of 16, uh, they were getting brute force for passwords. Um, and uh, so maybe you can kind of explain what we do there. In that situation, I noticed it looked like we had actually kind of proactively notice that and then contacted them and said, hey, you may want to contact these folks and make sure that these accounts are compromised because we can actually see what accounts they’re trying to hit and brute force the passwords for it.Arman Zakaryan: 00:24:24 Uh, yeah, so, you know, we, we have some specific hardening around the most sensitive parts of a wordpress site. If someone is trying to brute force a password over and over again, and they feel enough times they’re going to get hit with a Captcha, uh, it just so that they can prove that they’re human and not a robot. Um, and then, uh, if they go over a certain rate, they get rid of limited. You know, we’re also, you know, we have, we have human beings that are watching, uh, and reviewing the logs on a site. So like, if something catches our attention, we’re going to be looking into it a very, very deeply and reaching out and seeing if there’s any, any additional measures that we need to take. Most of the time,Arman Zakaryan: 00:25:10 the,Arman Zakaryan: 00:25:12 the what we call press armor layer, all the hardening that we, uh, that we apply it, which obviously I’m not going to share specifics, but yeah, most of the time the press farmer, we’ll, we’ll keep things at day. Uh, it’ll either stop something from happening or something does happen. It’s going to notify us and we’re going to fight you. We’re gonna help you remediate the problem. Sometimes websites do get hacked. It doesn’t look like it got to that point for top class factions. So, um, but yeah, it’s mostly the captions and the rate limiting. We’ll stop it and something does happen to get through. We have a daily malware scanning that will catch anything suspicious payloads. We’re always updating those signatures. And then, um, we do also have some virtual patching that we apply at the gate. We layer before someone could even make it to wordpress, so that, that kind of keeps us certain level of sanity. But just to know that some Middle Michelin oils, just malicious requests, uh, won’t even make it to your wordpress out bill. They’ll just stop at the gateway layer.Sean Tierney: 00:26:27 And Scott, I would imagine you guys, given the nature that you’re, you know, you’re, uh, how do you say this? Like the, you have the potential to piss off a lot of, uh, big Pharma brands or people, you know, they might not want you to exist because you’re proliferating class action lawsuit stuff. I would imagine that security has got to be really important to you guys.Scott Hardy: 00:26:49 Yeah, security is critical because we get so many, I mean, I get at least one or two seasons assists every single month from different billion dollar companies that are unhappy with us. And so we need to make sure that everything’s locked down. I mean, heck, we cover data breaches as in the class action lawsuits that are connected. So we don’t want to be the case where it’s top class action and we get hit with a date of brief data breach class action by one of our clients because somebody got in. So we are super conscious about everything. So that’s, you know, and I think it helps too that I am a nerd, huge geek. I mean, you and I met at Tempe geeks back when that was the thing here in Phoenix. And so even when I first started launched top class actions, making sure that we’re secure and had additional layers in place. Two to make sure that we don’t get hacked. It was huge. I mean, I was a hacker back in the nineties back when it was dial up at war dialing. That was me. So I’ve got a very sensitivity two, keeping the website secure and you know, we get thousands of submissions from viewers every single day and we need to keep their data safe.Sean Tierney: 00:28:08 Yeah, it’s personal. Well, we’ll leave it at that in terms of what we can talk about on the security front, but I think it’s clearly important to you guys, so I’m glad we’re able to help in that regard. All right, I’m going to jump to the next one. This is an interesting theme. It’s presented a couple of times. Um, this is, this was related in March of 16. So same month or shortly after you guys installed, uh, easy digital downloads for commerce. And it sounded like because that plugin happened to use a session cookie than that re initiated this whole cache issue or suddenly cash wasn’t working. And so that created a spike. But in our mon and I were talking before we got on here, what’s interesting is because we’re a host and we get exposure to a lot of these, we tend to have a relationship with the plugin authors themselves.Sean Tierney: 00:29:02 So gravity forms and easy digital downloads. And so we can actually reach out to them directly. And uh, in the, in the case as it was with this one, able to work with them. And so in this instance, easy digital downloads was able to issue a patch to their pro, a plugin, which allowed it to filter out certain traffic. So like if out of a hundred people, only one person actually needed that cookie that was being set for all hundred. And so they issued a, uh, an update, which then gave us a setting to say, okay, exclude those 99 people and only use that cookie on this one who actually needs the cookie based on a button or Amman. Is there anything else in terms of that fixed that you wanted to add?Arman Zakaryan: 00:29:48 Uh, no. So the fixed in these situations is usually very straightforward. The troublesome part is like you said, having that line of communication and that’s one of the benefits of working with a managed wordpress hosting company. Uh, Paisley is, uh, obviously we have some good relationships with these plugin authors. We actually host a, some of them ourselves. Uh, so you know, if you just break down those communication barriers, it’s really easy to get some effects. We can say, Hey, this is what we’re seeing, uh, for one of our clients. Um, this is how our platform works, which is just using standard page caching methods. Um, and this is what we think would be great. Uh, if the plug and work this way, and usually they’re very receptive and they say, Oh yeah, here, that’s going to go out in the next update. Or here’s a WHO’s a prerelease patch that you can apply on the plugin. And now, now it’s working a lot better.Sean Tierney: 00:30:48 Yeah, well they want it to work there. They’re motivated, well aligned. They went their own plugin works. That makes sense.Arman Zakaryan: 00:30:54 Cool. I mean, the, the things that we usually try to get fixed or are things that will help you on any, any hosting solution that that’s using a caching layer, whether that’s Mark Nash for engine x, Anything that respects the cache control and set cookie hetero values. Um, you know, those, those same fixes will benefit you anywhere else. Cool.Sean Tierney: 00:31:18 Um, all right, well I’m going to limit it to just maybe two or three more issues here in the interest of everyone’s time. Uh, it looked like a month later in April of 16, we, there was an issue with SSL and cloudflare and so maybe just explain what cloudflare is to those folks who are not familiar with it. Uh, Scott was using it on his site and so maybe you can talk about what that was.Arman Zakaryan: 00:31:44 Yeah, so cloudflare is a content distribution network. It’s also has a focused on security and preventing attacks for your website. So most of the benefits you get from that as you get the wide spread of their CDM. So if someone is in a different geographical location from where your servers located, it’s going to benefit from the caching. And, uh, they have some nice features like if you want, if your website’s under attack, you just go into cloudflare and say, I’m under attack. Uh, and then it will put up some additional verifications and things. So for visitors, um, somewhat like a little interstitial page, but it helps to verify that that’s a legitimate visitor before it sends you through the west side. Um, but I think the, uh, the issue was basically just a simple configuration problem with, uh, she was just causing a redirect loop.Arman Zakaryan: 00:32:42 Um, there were no, there was SSL cert four, um, the top class actions.com, but not for www I believe. Um, and we were, we turned off the forced SSL in the meantime and switched off flexible setting on cloudflare just to get the site back online. And then we took a look and fix, uh, fix things up so that it would, it would work correctly over the full SSL mode. And then we turn that had them turn that back to uh, to the full security mode. Um, anytime you’re using a proxy service like five flair or fastly or what have you, uh, your website visitors are visiting that company’s servers and then that server is acting as a proxy or a broker to communicate with your server. So even if you have SSL on, on, on cloudflare, if you don’t have SSL on your origin servers, that’s a somewhat of a security issue. Uh, so getting that to work sometimes where it’s SSL all the way through the whole system, uh, can cause some problems sometimes. Um, just about having, uh, having some of the staging environment up first and testing it out there and um, making sure that you do have https, uh, at, at every point to keep all the communication secure.Scott Hardy: 00:34:10 And Sean, something I’d like to call out here is that we had, with all the SSL prompts we had back then, it was nice when I got an email from Pagely just this month or last month saying, Hey, we notice your SSL certs expiring within the next 30, 60 days. Whatever it was, what do you need us to do to help you out on that? And I went back and forth with them a couple times on support and they gave me what I needed and I went and got my SSL cert certify. It took me all at once. I knew what I needed. It took me 60 seconds and I sent it in and you guys got it updated. So it’s nice that you guys are on top of those types of issues because you know, I’m, I’m, I’m busy running the company and I don’t necessarily pay attention to that. Last time I think it just expired and then I flipped out and all hell was breaking loose and said, you guys actually gave me the heads up and made sure it was done weeks before it actually expired. So good work on that.Arman Zakaryan: 00:35:06 Cool. Yeah, no. Doing that outreach for, uh, for expiring SSL certs. A style, you’re actually on a long time customer, uh, you still on our stack, which, uh, we’re replacing everyone, uh, with the new gateway we call areas. And one of the huge benefits to that is let’s encrypt, just being built right now for those who are not familiar with let’s encrypt is free way to get an SSL cert. Um, there are short duration, sir. It’s usually 90 days. And uh, do you have to basically keep renewing it a two, then you got free Ssl as a result, you just have to keep renewing it. So what we’ve done with our next generation gateway, because we’ve built that straight into the whole system, uh, there’s a new, uh, there’s a new atomic a web interface where you can request SSL certs, it’ll apply them automatically for your APP.Arman Zakaryan: 00:36:12 Uh, it will keep them renewed, you don’t have to do anything else. Uh, and then with that same system you can manage your own SSL cert. So if you still want to have a paid search or wild carpets or uh, you can just import it from there. And then new system also automatically sends those emails out when they’re expiring. So, uh, we are sending out emails for any customer if we are noticing that their sites as the cell is going to expire, a sort of like we’ll do that manually, but the new system does it automatically, uh, for every, every hosted site. Um, so definitely a, we should get that upgrade on the books for you soon, Scott, so you can, you can benefit from the areas gateway. Uh, it has, it has that, uh, let’s encrypt cert. So you even have to go and buy a new surgeon in the future. You can just have it for free and then he’ll renew automatically. Um, and there’s a lot of other benefits to areas as well. No, on that one, the premium metal shelves, we have the fancy bar at the topics without her stuff. Oh yeah. If you want the bar, you’ll have to get the, the paid search. But uh, it will be self service. You won’t even have to contact support to get that done. Let we just go click around. But obviously we’re happy to do it for you as well. It’s all about the bar. I’m high maintenance, you know, then, uh, okay, let’s see here.Sean Tierney: 00:37:33 So if another theme, Armando was going to bring this up because this could be relevant to other people who have this, but, uh, April 16, they had some slowness issues and we had to determine it to be related to calls to Admin Ajax. Can you talk about what that is and what we do? I know we have some ability there, you mentioned with get requests and caching and whatnot, but can you maybe just talk a little bit about that?Arman Zakaryan: 00:38:01 Yeah, so just a quick intro to amend. Ajax is an end point in word press. It’s been there for a long time. It’s used for all kinds of things. Um, you can either make a post request to it or I get requests and obviously if it’s a post request, you can’t cash it. But if it’s a get request, you can, uh, as long as you’re not sending cache busting headquarters, uh, like the sec cookie or the cache control Max stage zero or things like that. Uh, so I think the issue with that, a slow site event in April wasArman Zakaryan: 00:38:45 yeah,Arman Zakaryan: 00:38:47 who’s every request basically invoking a call to have an Ajax. And it was returning, um, image URLs that have spaces in it, which was causing a bunch of redirects to happen and for fours. Um, so I, I believe, I believe we just kinda jumped in and made some emergency ashes to, to get things back to normal.Arman Zakaryan: 00:39:17 MMM.Sean Tierney: 00:39:18 Yeah. My take on what happened there. Yeah. This thing with the image spaces, it sounded like Scott, you guys had a, like a, a special plugin set up. So any four fours would bounce that person to the homepage and there was something happening where in it didn’t like the fact that there were spaces in the image names. And so those calls like five of them on a page, you then get like five redirects to the homepage happening in the background. So it was like amplifying each page request to like times five or at least that’s what I made of it. MMM. Yeah.Arman Zakaryan: 00:39:56 Yeah. So this is, you know, this is one of those situations where it’s good to have a managed wordpress hosting your site because they don’t have engineers that will, we’ll be able to understand what’s going on and, and secondarily, like have, have the ability to engage and make changes on your behalf in an emergency. You know, if, if you’re just sort of staying on your own, you know, you grab an AWS accounts, spin up a server and run your site. The second something you don’t understand happens, you know, you’re kind of on your own or you might, you might have to reach out and then try to find somebody who can, who can assist you with thatArman Zakaryan: 00:40:37 problem. Um,Arman Zakaryan: 00:40:39 so, you know, that’s, that’s just stuff that you get from being on page leave. Like all our engineers work on wordpress all day long. We’re always, uh, fixing this or that problem for our customers. And one of the times, you know, it’s really easy for us to spot, oh, hey, this thing is the thing that’s causing your problem and here’s what we can do to fix it. Or, here’s what we did as an emergency patch just to get you back online.Arman Zakaryan: 00:41:08 MMM.Arman Zakaryan: 00:41:09 And you know, we, we always, we always try to sort of say like, Hey, this is, these are the exact things we did. This is why we did them and this is what we recommend as a long term fix. Um, you know, some of those fixes or just having a staging environment where you can test things out first. It’s really, you know, a simple mistake, an extra space somewhere can cause all kinds of problems if you have enough traffic happening on the site. Uh, so, you know, we make it pretty easy to uh, to create a staging sites. We have Beasley sink and we have the clone tool, uh, that makes it easy to, uh, to have a testing environment that’s running on the same server running on the same hosting stack that your, that your life sites on a, so, you know, those types of things are, are helpful for preventing a simple problem, like an extra space from causing a larger problem in production.Sean Tierney: 00:42:10 Cool. Scott, I was going to ask you, what can you like, just give us roughly or whatever you’re able to quantify when one of these newsletters goes out, what’s that worth to you? Like downtime during one of these critical periods where now 700,000 subscribers just got your email. Like can you, are we talking thousands of dollars an hour? Well, what is the uptime worth in that scenario?Scott Hardy: 00:42:37 Well, so oh, more traffic. Um, 41% of it is organic search and 28 and a half percent of it email. So you know, 30% of our traffic is generated by that newsletter. So being able or, or having going down when the newsletter goes out, huge in terms of the submissions and you know, and that’s, and that’s how we get paid. And so we, if we see if we’re down and our submission rate goes down, but our clients cancel their advertising campaigns because our business has made on being able to deliver now we don’t have one subscription rates. We don’t sit there and try to tie people into your loan agreements. Okay. If we don’t produce then you can cancel. And you’re only with us for a month. And so it’s critical that we don’t have any downtime because especially when we’re talking about some of these case values, these case values could be literally $1 million for some of these clients. And some of these campaigns, they might sit there and run with us four a long time. Um, especially when you’re looking at the pharmaceutical drug medical device towards that person is going to hit multiple websites. Typically they’re going to submit their information to us and they’re going to submit it to other people.Scott Hardy: 00:44:03 And so if they try to submit it to us and it doesn’t the work, then they’re gone forever. And you know, our client could miss out on a six or seven figure case and then we lose, you know, upwards of anywhere from eight to $10,000 a month in revenue from just just one case.Sean Tierney: 00:44:23 Yeah, yeah. So it’s imperative that you stay up during those at any cost basically.Scott Hardy: 00:44:32 Yes. Yeah. That’s where I get, I get pitched by friends or old associates with other hosting providers and I’ve got friends that say, Scott, let’s just go ahead and move you on to your own Amazon stack. [inaudible] is built on Amazon. What do you spend all the money you’re spending on it? It’s pretty simple. I can’t go down and I don’t want to have two or three wordpress engineers that are super experts that I’m spending, you know, three or $400,000 a year in salaries and benefits on when I can pay page lead to as these experts on it. 24 seven that if something breaks, usually you tell me and I’m not sitting there freaking out. Cause typically as soon as someone goes, Scott, something’s broken, a lot of times already have an email from you saying, hey, we’ve, we saw a, got an alert, we’re looking at it and we’ll get back to you within a couple of minutes. Um, so yeah, it’s, it’s critical. And that was always when I, when I built things up since I,Sean Tierney: 00:45:28 Yup.Scott Hardy: 00:45:28 I know enough technically to get in trouble, but I don’t know enough to log into the server and take a look at who’s hitting it and, and you know, I lost those skills 15 years ago when I went into management. So it’s critical that we’ve got a team like you guys behind it that can sit there and look at the server and see exactly what’s happening at any given time and be able to address it.Like this content? Meet Pagely.Sean Tierney: 00:45:53 Yeah, absolutely. I think there’s another benefit too. So that’s one for sure of having, uh, you know, have it be an op x versus a cap x. You can just pay a monthly fee and you know, that it’s handled versus having to have like the overhead of salary and whatnot. Not to mention the 24, seven thing, but I think the other interesting thing is that like our mom says, this is just a day in the life for us. You know, we see it’s hitting sites and hacking attempts and downtime and whatnot. It’s like we deal with this on a daily basis and so we have this broad exposure to all kinds of attack vectors. And things that happen that I think, uh, you know, an n of one, a single developer that’s on staff with you, they see tunnel vision of what they’re seeing on your site. But we have that, that benefit of seeing across, you know, thousands of sites, what’s happening and what the design patterns are of what works and whatnot. Okay. SoScott Hardy: 00:46:46 cool. Yeah, and just having, I just switched from my earpods just by having that breadth of experience that I know you’re not just seeing these prompts on my website, but you’re going to see on, you know, the 100 people that you work with. So that’s the confidence and that’s why I don’t switch. I’m not going to switch to save $1,000 a month. Doesn’t make sense. I can, I could lose that and five minutes of downtime.Sean Tierney: 00:47:12 Okay.Scott Hardy: 00:47:13 Yes. Let’s go ahead and stay up and I’m happy to pay a premium when I know that stuff gets taken care of swiftly. And then I don’t get gals, like you said, the guys that were sitting there saying, hey, go pay somebody else. You know, 20, 30, $40,000.Arman Zakaryan: 00:47:29 The code updates when it was just Armand spending 10 or 15 minutes looking at it going, oh, it’s common out these lines of code and you’re fine. That’s huge.Sean Tierney: 00:47:38 Yeah. Yeah. Well, and the other thing is, because we have your live traffic and we have logs and we have empirical data that shows us where the issue is. I think it’s one thing to get pitched by a guy who claims he can rewrite your site and optimize the code and make it run better. But it’s another thing to have the actual code running in the logs and everything to be able to diagnose it and say, Oh yeah, we just, we see this every day. We know that these are the three lines that need to be common. So.Arman Zakaryan: 00:48:05 Right. Exactly.Sean Tierney: 00:48:07 Cool. All right, well there’s other stuff we can talk about, but I think in the interest of time, I wanted to leave a little bit of room at the end because we, you were actually the very first customer to do something, uh, that we dabbled with, which is this idea of what we call a [inaudible] performance audit. And so we went through that. Armand, would you be willing maybe to just chat through some of the things in that? So we, we, we, what we did is we produced a 14 page audit report essentially of, you know, things that we see, opportunities for improvement. These are obviously just, you know, recommendations. Some of it is just verifying that you’re good on things like security, but we’re able to suggest like future enhancements if you want to spend on it. Here’s where we see the opportunity to make the site run faster and whatnot. Um, so Armando know if you’ve got it open, maybe you could just kind of step through a couple of those.Arman Zakaryan: 00:48:53 I’ve got it over here. Uh, so yeah, the press performance audit is basically kind of taking all the things that we recommend on their knowledge base about, uh, making sure your site’s caching a common areas for improvements. You know, the most common things that could make a wordpress site slow. And, uh, we, we kind of follow our own advice and go through that, uh, for your website. Um, and at the end we compile a report, uh, that’s easier to digest and something that you can hand to a developer to start to start making some fixes on for you. Um, and you know, the type of things we cover in that report is seeing if your site is cashing well, I’m seeing if there’s any excessive cache purging happening, which can also hurt the cachectic rate. Making sure that your site is probably using, uh, our press CDN.Arman Zakaryan: 00:49:45 Uh, cause that’s the most efficient way to get the static content out to your visitors and it, the CDN has a higher allocation for your plan versus the bandwidths out the server. Uh, we do a security audit, uh, just to make sure if there’s any glaring issues like an out of date plug in or too many administrators accounts, you know, we kind of recommend, uh, we make some recommendations there. Uh, as well as a recommendation for like two factor authentication, things like that. Uh, we review the site plugins in use, uh, for any plugins that may be causing large queries to happen or slow berries. Uh, we were at, we make recommendations on optimization. Is there, um, and then, uh, yeah, it’s if there’s anything specific about the uh, the customer that they came to us and said like, yeah, give me the press conference report and here’s my actual concern.Arman Zakaryan: 00:50:46 Like this is my main concern that I have. Uh, we, we keep a section in that report to really dig into like that specific concern that they may have that may not, I may not be covered by the standard things that we look at. And so, uh, for top class actions, the thing that that was the main concern was like back in slowness. And what we found when we looked at that was actually a pretty common issue that happens in wordpress. It has just large auto old options and for those aren’t familiar a auto load options are things in the database and your wordpress site that if they’re set to auto load, yes, then it loads that data on every request. Whether or not the request being made needs that data. And so if you have something like a 30 megabyte autoload option, that’s going to obviously have to do a lot overhead to, to get that data and process it, uh, before it can serve the request.Arman Zakaryan: 00:51:46 Um, so just cutting down on those auto load options usually helps with that. Can slowness. There’s also a couple of things. Um, I think just sort of related to, to the footprint of the top class actions say that, uh, some clients could use a couple of tweaks here and there basically. Uh, so it’s not loading as much data on certain pages. Um, you know, things like having a lot of WP contests can also take up PHP workers. You know, we have some solutions there. We can either set up the WP miniums plugin made by 10 up, uh, which utilizes a gear man backhands to get a synchronous job, queuing a, or we can enable inland prime mode, which keeps your regular piece be workers from having to be stuck hailing a chrono tasks, uh, things like that. Basically the rest of warm his report, you know, if, if someone, uh, just reviews like what is on our inner knowledge base, uh, as far as what we cover and what we recommended there, you know, they can come to all these conclusions on their own. Um, the real value of the press performance report is we do all that leg work for you. It’s sort of a white glove service where we’ll just put it all together for you and make really clear recommendations on the problems that we find. Cool. Yup.Sean Tierney: 00:53:16 And I think also our mon, you were mentioning like we do cost optimization in when we can. And so like a good example of that is Scott, you were at point having to run on an edit one. Ha. So a fairly beefy ha pair. I think right now you’re currently on a vps to ha. Uh, so we were able, you know, through some of the code fixes that we did and some of the, your developer has been able to apply a tune it to the point where it can actually now your traffic has grown and yet you’re able to go down to a smaller instance size because it is caching. Well at this point.Scott Hardy: 00:53:50 Yeah that was, that was Greg Smith. We’re looking at, we all are always trying to optimize our expenses and not overspend. And so yeah, it was nice when we can talk to you about that and figure out ways to save some money without, without shaving performance. Google and organic traffic is so huge. Remember, remember we look at it, uh, 41.4% of our traffic is organic from search. So we need to make sure that we are swift fast and highly optimized all across the board. Otherwise we’ll get penalized. In fact, we just found out we, we’re number one and two for a class action settlement on a beautiful yeah. And we’ve been fighting to get back up there cause we were a couple of years ago. But with so much competition out there, everything, everything has to be optimized. And the press performance before it helped us tweak things to get a couple of microseconds faster. Couple of microseconds makes a difference. Google.Sean Tierney: 00:54:49 That’s awesome. Well congrats on all the, uh, your continued growth and being back on top and number one there. Um, is there any words, any like parting thoughts you have for someone maybe who’s listening, who has a high traffic site of their own or do you know they’re on the up and up and they’re growing and they’re an entrepreneur? Uh, any just words of wisdom you have for them from your experience now having been through this for quite a few years. Okay.Scott Hardy: 00:55:16 You know, I think it’s when you’re growing your website, especially if you’re in a position like I am where you know you’ve got either a little technical, know how or not at all. You want to make sure that you were really working with experts in the fields and when you are growing, you want to be spending money on the right things. Now, if I had taken some salary money and put it towards the technical side and said, okay, I’m going to hire a senior level wordpress engineer to optimize why website and handle it, it, it would, it would have been a very poor decision, especially early on because that money, I needed to hire writers and go into digital marketing. MMM. I needed to put that money in places that could help the revenue growth. While I could depend on a partner like Pagely to support technically and you can’t, you just can’t one expert into a business and have them even if they are extremely experienced, have them have the breadth of knowledge that you need.Scott Hardy: 00:56:33 When you guys deal with hundreds of different companies on a daily basis. And my guy just feels one and we know we had the same problem with, with our SEO side where one of our developers said, no, you’re great. And I happened to be at a conference talking to an SEO guy out there. I said, hey, can you check things out cause I think that our growth is a great, and he said, yeah, no, you’ve got this problem, this problem, this problem, this problem. And that’s because they have, they’re working with so many different clients that they see all of the issues. So my advice would be no higher a partner that can support you technically and isn’t just sitting there saying we’re going to keep you up. But they’ve got the ability digging, dig into your issues and fix them and not nickel and dime you.Scott Hardy: 00:57:26 I mean that was an issue that I was having when I was working with one of the huge providers here at Phoenix as they said, yes, we’ll go ahead and support your wordpress environment. And by the way, you need this professional services agreement that was triple the size of the hosting agreement. Okay. And we’ll won’t be able to give you some support while with Pagely. I’ve got the experts in there and you know, we, we pay you a fair amount every month, but there’s, when Armand Armand fix that problem for us right up front, honestly I was expecting a professional services bill from you guys. Yeah. You know, we, we build on it. Developers know around $50 an hour or whatever it is. And you guys don’t do that. You guys have said, hey no, we want you up and we’ll take care of it and you’re not trying to charge me an extra 150 $500,000 whatever it is a month for some of these odd requests that come through.Scott Hardy: 00:58:22 You just take care of it. And that helps lower the stress. Well, stress levels as a client and that adds a lot of, a lot of, a lot of value for me. So that’s my advice to ontraport. Entrepenuers is just don’t necessarily go out and hire the super techiest most expert guy you can get look at your money and see if you can bring that in to a partner that will, that will handle it. So you can focus those big dollars on something else to help you actually grow your business and not just support it. Does that make sense?Sean Tierney: 00:59:00 Yeah, absolutely. Cool. Well, Scott, thank you so much for taking the time today. If people listening, uh, feel they might be leaving some, some class action money on the table and they want to use your service, what’s the best way for them to go about doing that?Scott Hardy: 00:59:16 Just going to talk class actions.com. Sign up for our free newsletter. You’ll get an update from my people. You get an update from me personally every single week. Just talking about how things are going. We make sure that you don’t miss out on anything. We are 100% free for consumers and overcharged consumers, dime access our website and find literally thousands and thousands of dollars a month or a year that you’re missing out. No, I just reported we should put it on some different settlements that payout up to almost $4,000 per claim, uh, for some different automobile related settlements. And you know, if you get spam phone calls, you hate those come to us. We have class actions that cover it and those typically pay up hundreds of dollars for per claim. So it’s one of those things if you need cash, what are the top of class actions.com sign up for the free newsletter and be persistent and be patient because these things don’t pay up quickly. They take months or years to pay out. But if you consistently submit claims, I have viewers that pull in four to $10,000 a year from claims that they’re legitimately claiming no, just by paying attention.Sean Tierney: 01:00:25 Yes. I go throw some lines in the water now, sit on them and get some cash. Few months. Exactly. Exactly. Awesome calls a day. So I might go stand up. I was just thinking the same thing and I’m, I’m, I’m even on the do not call registry and using Google’s block on Google voice and I’m still getting a bunch of Robo calls of a mini to sign up myself.Scott Hardy: 01:00:47 Yeah, you should. I mean, I’ve got my interviews that’s coming up in about an hour. Uh, we’re going to be covering four different new TCP lawsuits that are all, that’s the telephone consumer protection act. And those are all settlements that are done by larger companies that are just spamming for text messages or phone calls. And there, you know, those things pay out anywhere from 50 to a couple hundred dollars per claim. And so if you can, if you’re getting annoyed that much, then by all means get your information and get that money back in your pocket. So you’re like, yeah, you called me, you spammy. But now I’ve got 200 bucks. So neener neenerSean Tierney: 01:01:24 cool. All right, well I’ll let you go. Thanks again so much for the people listening that you can come back to the blog. We’re going to be putting these things up, hopefully doing a series of this. And so let us know what you think. If you found value from this, leave a comment and, uh, we’re happy to chime in and answer your questions. All right, everybody.Scott Hardy: 01:01:41 Thank you.March 27, 2019April 2, 2019 Video Case Studies Author Sean Tierney Sean is Director of Sales for Pagely. He now lives and works in his favorite city Lisbon, Portugal having graduated from Remote Year. Sean is an avid guitarist, kite surfer and practitioner of Krav Maga. Follow his travels abroad on his personal blog scrollinondubs.com, on Instagram or on Twitter and tune into his podcast for nomads. 0 Comments Cancel reply Comment:Name * Email *Save my name, email, and website in this browser for the next time I comment.