Video Case Study #4: GiveWP

Sean Tierney: 00:01:19 All right. Hey everybody. Welcome to the video case study number four. I’m here today with Devin Walker. Devin is founder of impress.org and has grown into a team of more than 15 employees within two years of incorporation. In total, his software has been downloaded several million times and has accumulated more than 350 5-star reviews. He’s best known for his creation of the GiveWP, online fundraising platform built for WordPress. His work is being used by millions of websites and has been featured in Product Hunt, WordPress Tavern, Torque and a couple other prestigious publications. He’s an active community contributor through his code contributions, his WordCamp and meetup organization and speaking efforts, as well as fostering and supporting popular online groups like Advanced WordPress and WordPress for nonprofits. Welcome, Devin.

Devin Walker: 00:02:07 Thank you.

Sean Tierney: 00:02:10 So I think we should start with GiveWP since it sounds like that’s really the one that you’ve had with us for the longest and we’ve got a lot of stuff that we can talk about on that one. First, can you just start by explaining what is GiveWP?

Devin Walker: 00:02:24 Sure. So GiveWP essentially is an online fundraising platform that lives inside your WordPress website. You can do everything from, of course, set up multiple donation forms and campaigns according to, you know, ongoing fundraising or event-based fundraising. For instance, coming up now is a Giving Tuesday in the holiday season. So, you know, you can schedule a goal to kick off and then you can also manage your donors and integrate with, you know, more than 15 plus payment gateways right now. It’s basically everything you need to fundraise if you’re a nonprofit or cause or simply looking to raise money for some, you know, event in your life. So, yeah, that’s, that’s definitely our premiere plugin. We work on it every day and our whole ecosystem of add-ons and the website, it’s all hosted on Pagely.

Sean Tierney: 00:03:26 Yep. Cool. And so this is distributed software. It’s plugins that people are downloading and as best I can tell, I read through your support history of about 69 tickets I think was the total and most of the issues that I could tell stem from the fact this is calling back to your licensing server. And there was various stuff related to that; that fact that it was calling back and were getting some issues there so we can talk through that. I forgot to introduce Arman. Arman Zakaryan is our Head of Hosting Ops. He is on the call as well. Welcome, Arman.

Arman Zakaryan: 00:03:59 Hello.

Sean Tierney: 00:04:01 And if we have any bells or noises that you hear in the background. Arman had a fire drill going on a minute ago, but I think it seems to be stopped. So hopefully, knock on wood, we’re good there. But let’s start out. So you guys were provisioned in it looks like September 15th of 2017 was when the server got set up. And then at ticket that I’m looking at five days later where we’re talking about, this is where the Easy Digital Downloads, which is the system that you use for your licensing, it sounded like the end point was just getting hammered. And Arman, are you able to talk about what we did in that situation with the rate limiting?

Arman Zakaryan: 00:04:41 Yeah. It’s actually been quite a journey over the last couple of years with this end point. So when they first joined Pagely, we hadn’t yet been using our new Ares gateway, which is our take on using NGINX with what customizations to make it easy to apply different rules and better rate limiting and stuff. So when, uh, when Devin’s site first joined Pagely they were on our previous gateway and um, that was more in line with a pure NGINX, you know, sort of config style. Um, so the initial issue I think was simply that the real IP configuration wasn’t set up right cause they go through security, uh, for their firewall. Uh, so that’s just some NGINX configurations, uh, to set up the real IP. Real IP is basically an authorized list of IP addresses that are allowed to send an export For-Header so that your origin server knows what the real IP of the visitor it is.

Arman Zakaryan: 00:05:53 So you don’t have that set up right then if there are any limiting rules or anything that applies changes or policies on a per IP basis, it’s going to lump all of those requests as coming from the IP of your sending proxy, not as, uh, an individual IP address per visitor. So that was just an initial issue out the gate, just getting the real IP set up correctly so that we could bucket each visitor the right way. Um, and then once we got that sorted out, there was definitely a high volume of traffic for that end point. Uh, so the, the rate limiting was not, uh, miss mislabeling traffic anymore as coming from a single IP but there was still a lot of, a lot of requests coming in overall. Um, and I think most of this, most of this, uh, case study we’ll be talking about the different ways

Arman Zakaryan: 00:06:53 that issue was a worked around and worked on from both sides, from both Pagely, uh, and with, and with GiveWP. Uh, there was lot of effort but into, uh, making changes on both ends within the plugin itself as well as on the hosting infrastructure, uh, to sort of bring that under control. So, um, if you want, we can dig into more detail on that ticket or we can, uh, we can move on through up throughout the rest of the time line.

Sean Tierney: 00:07:23 Yeah. Well, I, I’d just like to get Devin from your perspective at this point, you know, you’re five days into hosting with us and it seems like you’re getting real rate limited. Uh, you know, our system thinks that all this traffic is coming from one IP address, uh, given that we weren’t passing that header through. So what is, once that’s solved and now it is clear that there’s still some other issues, um, what are you guys thinking at this point? Like, are you thinking about code changes? Are you, are, you know, what’s going through your, your mind basically at that point?

Devin Walker: 00:07:58 Yeah. So, you know, the reason why I moved over to Pagely is because we aren’t experts at hosting at all. I’m not a system engineer at all and you know, know barely enough to be dangerous in that area. So we weren’t getting any answers from our previous hosts about, you know, what we can do to mitigate that as how we could possibly, uh, you know, limit some of these requests coming in, throttle them, whatever term you wanna use. Um, so when we moved over to Pagely, I knew, uh, you know, we’d get some more answers and we definitely have. But what I was thinking was, you know, it’s just the first little hurdle to get over. Uh, I knew this wasn’t going to be a quick fix. And, um, it’s definitely a lesson for, uh, you know, those trying to build a large business around a distributed plugin where they’re, you know, they’re checking, they’re calling home from tens of thousands of wordpress websites, getting this end point, activating licenses, deactivating licenses, checking subscription statuses.

Devin Walker: 00:09:02 Um, sometimes it works. Sites out there have misconfigurations where they’re checking updates. Um, and then extremely high frequency. And, uh, you know, some of those websites are, um, owned by people who don’t even know this. Um, so we try to reach out to them and help them out as much as we can do that. But in the nature of, you know, how many customers we have that can be difficult. So I got some really good answers and insights from the Pagely team. Some good ways to troubleshoot on our end. Um, we have an internal doc that we have been, um, compiling with ways to, you know, troubleshoot within our, um, server environment. Um, but the initial thought going back to your question was, um, you know, I didn’t really understand the whole issue to be honest. I set Real IP, I get it now much in two years later. Um, but it was good to get that attention and that help and uh, and faster response time from that support. That’s really the best thing about Pagely. It’s like, not only is the infrastructure great, but I mean, I, I submit a lot of support requests for things that most 99% of other WordPress hosts not even think about even supporting at all.

Sean Tierney: 00:10:26 That’s awesome. Was there any takeaway like in retrospect and knowing what you know now, is there any lesson or any, anything that you wish you could have done differently in terms of how things were architected or any take away about the calling home mechanism or is…

Devin Walker: 00:10:44 Yeah, no, I mean, we now are just finishing up our licensing proxy server, which you guys have helped also redirect some of our staging traffic in our testing phase in the last couple a months for that. Um, and we’re getting ready to roll that out live. Um, it’s difficult cause when you’re launching a first product, like, you know, we launched this in 2014, early 2015 no, we didn’t really think about how over time, four years later, the growth of it could really cause this to happen. And I’ve spoken to a lot of other WordPress, uh, successful WordPress, um, owners out there and they had this same issue too. Um, and they’ve developed their own work arounds, um, as well. But the thing is it’s such a niche type of problem to have that there is no other really solution other than, you know, not using that, that product to sell your licenses or rolling your own mechanism. So

Sean Tierney: 00:11:43 I mean it’s a fortunate problem. I would say like you did it the right way. Like it would be pre optimization to worry about scale when you don’t have scale yet and you know, to spend a bunch of cycles, something that’s not even a problem yet. So it just seems like one of those things that you’re lucky to bump into it cause it means you’ve done something right with your product that you’re getting that much usage.

Devin Walker: 00:12:02 Well, yeah, we, you know, we have this newer product called WP business reviews. It’s not even near the scale of GiveWP. And I’m just thinking in my mind, you know, three years from now where we’re, we’re gonna be much better prepared since we already have the technology almost rolled out for GiveWP and we can kind of use that similarly. Um, but I know it’s, it’s a good problem to have and hopefully we do have that problem and a couple, you know, you’re saying we’ll see that better.

Sean Tierney: 00:12:30 Yeah, for sure. Um, all right, well, so let’s maybe Arman, unless you have anything else to add on that one. I was thinking we’d go to the next one on October 20th the Dad, I have just one quick

Arman Zakaryan: 00:12:42 I have just one qiuck thing to add. Um, so just, just to close the loop on that real IP thing and with, with popular, uh, firewalls and CDNs out there like security or Cloudflare, CloudFront, uh, our new, our new Ares gateway actually sets up all the real IP stuff automatically. So that’s a lesson we learned ourselves from a previous iteration of our product. We applied it to our new products, which automatically sets off that real IP stuff. So that just leads to less, less problems that they gave for new customers. Now that we have that automatically being configured. So if you are sending through Securi or any of those other big popular ones, um, you know, we have that, that part of it already set up out of the box.

Sean Tierney: 00:13:30 Can you, while we’re on Ares, can you just, what is the gist of the change with areas? What does it do? What does it allow us to do differently versus the old gateway? And I know it’s a lot, I know you can’t just like wrap that up into a sentence, but like what’s the kind of the crux of it?

Arman Zakaryan: 00:13:45 So really the big benefit to customers is any sort of customization that you need at the, at the web server layer is not a difficult thing for us to do. We have, we have systems in place and workflows in place and an entirely customized way of expressing those rules on our side. That makes it really easy for us to set up anything that you need. So we’re not having to go into some random engine x config file and setting up a, you know, these tricky rules. And then having to put that into, some other configuration management system. You know, our Ares system has configuration management built in, is all tracked with Git we have automated tests that run against the rules before they’re uploaded, things like that. Uh, so we’ve, we basically built the system where the, the edge case is not really the edge case, it’s more of a stress case and we can really easily accommodate a custom settings that you may use for your site. That could be a subdirectory proxy that could be, um, could be custom rate-limiting roles. It could be go IP based redirects, it could be all kinds of things. Um, and those are, you know, it’s just another day in the life, uh, for us to set that kind of stuff up.

Sean Tierney: 00:15:15 Cool. Nice. All right, well, so let’s jump to, uh, October 20th there was a ticket about a, the downloads directory was caching and I, it sounded like Devin, you had some downloads and it was serving like antiquated versions instead of the newest versions. And then we discovered that that was caching and that Arman, that was, that was a fairly simple fix, right? That was just a rule to not cache that directory. Um, is there anything else to add? Like would Ares have made that simpler in any way, what else would you say about that situation?

Arman Zakaryan: 00:15:50 So this is something that you’ll run into when you use any caching layer. If you’re using a CloudFront or if you’re setting up Varnish or if you’re setting up the NGINX, NGINX is caching features. Uh, if you’re caching content, then you need a way to purge that content when you, when you actually change the, the, the content on disk or update a post on WordPress or whatever. Uh, so for any dynamic content as part of your WordPress site, your pages or your homepage are different categories, menus, things like that. Anytime there is a change that you make within WordPress, we have hooks that will automatically go and purge that from the cache so that you know, it’ll go and get the new copy. Uh, when you’re changing static files, obviously there’s no, there’s no hook to, to trigger cause you’re not doing that via your wordpress panel.

Arman Zakaryan: 00:16:46 You’re logging in over SFTP or you’re, you know, editing a file on disk. Uh, so if that is a file you’re updating frequently, then we can very easily set up rules to exclude a caching for that specific item. Um, another way around it is to add a query stream to the end of the URL. So my file.zip/r or version equals, you know, uh, some string. Um, so if you have a query string like that at the end of your URL, that’s considered a unique cash element, so it will go in and get a fresh copy of that file. So that’s another way to do it. Uh, and then the third way to do it is to make use of the, uh, the purge API that we have available. So if you do know that you uploaded a new copy of this file and you want to make sure that it’s, it’s clear from all the caches, then you can issue a purge caches button inside of the WP admin panel.

Arman Zakaryan: 00:17:49 Uh, there’s a whole section that’s labeled occasionally where you can, we can do those things and you can purge your CDN, you can purge the PressCACHE, which is the NGINX base cash that’s on your server. Um, and then we also provide a programmatic way to, uh, to, to use the purge API within your WordPress plugins. So you can make, you can make use of that and point, uh, in your plugin or you can just do a WP Cli call, uh, to purge the cache for a specific path or for everything. Um, but yeah, so what we ended up doing for this, uh, particular case was just to set that URL to not ache. Uh, just it’s the cache bypass basically. Um, so that’s fine to do if it’s a static file and, you know, I’m just gonna update this frequently. I don’t want to deal with purging.

Arman Zakaryan: 00:18:44 I don’t want to do with query strings. That’s fine. Um, we’re a little more hesitant to bypass caching on things that are dynamic for obvious reasons being, if you have a lot of traffic going to an endpoint and that’s running, you know, as part of your WordPress application or spreading some PHP, um, if that workload is really intensive then it can overload the server could overload your database. That’s one of the huge benefits to caching in general is to avoid the work for your application server. Um, but yeah, if it’s just a static asset, it’s totally fine to, uh, it’s not really a big deal for us to set up a cash bypass, uh, for specific things like that. Um, and so with Ares, how that’s easier is again, we’re not having to implement or express that rule in engine x sort of language. Um, it’s still using NGINX, but we have, we have our own rules engine where we can express it as a match and an action. So we expressed this to JSON, we can say match URL value, the URL is my file.zip. And then the action is set cachable – no.

Arman Zakaryan: 00:20:05 And that’s pretty much it. Um, I mean, so if you compare that to how you would be writing a rule inside of NGINX, uh, you have to take into account how all the location matches that are existing, work. You have to take into account how, uh, any rule that you’re, you’re adding it to match this, this file or this, uh, folder, uh, how that may impact approximating other requests for the rest of the content and that path if you’re doing like really conditional stuff like. You really, you, you have to know a lot of…

Arman Zakaryan: 00:20:38 like the inner nuances of NGINX, uh, to really set up those kinds of rules properly. Um, whereas, you know, we just have a lot of those actions abstracted out with our Ares rule engine. Uh, so the end result is really, it’s easier for our support team to let those rules. It’s easier for anyone picking up a, your account next time to understand what rules are active. Um, it’s just like easier to work with. We designed this Ares thing, uh, and the rules engine for this use case is for hosting websites, primarily WordPress, but it can support a lot more.

Sean Tierney: 00:21:18 Nice. Yeah and that readability, it sounds like it’s easier to implement but easier to maintain, which is a big thing. If people need to go back and you don’t need to wade through some esoteric kind of hard to read syntax, that’s gotta be a nice, nice feature of it. Um, okay. So let’s, let’s jump to the next one. And this one seems kind of like the, the bulk, like this was kind of the major meat of this whole relationship here, it sounds like was in this issue. Um, it sounds like, and let me tell me if I’m right here and Devin, maybe you can, I mean obviously you’re the one feeling the effects of this, but let me see if I can summarize it. It sounds like we were doing rate limiting, but we were getting, uh, like false positives. We were, we were, we were blocking out legitimate people that were just a, you know, they were starting to have licensing issues like there cause they weren’t connecting to your licensing server, uh, due to the rate limiting.

Sean Tierney: 00:22:13 And so the trick was really trying to calibrate that and salt, you know, make it work as best we could to keep the server alive without making you spend a lot of money on hardware just to keep it up while you developed a code fix. And it sounds like you, you were able to identify and work on a code fix, but we kind of did a bandaid solution where we, we, we actually threw hardware at it to make it work in the short term. Uh, and can you maybe just like take us back to there to that point, what was happening? What were your customers experiencing at that time?

Devin Walker: 00:22:46 Yeah, so there’s a couple different, um, moving parts here. Um, one, you know, going back to the way that, um, some of our customer’s sites were just hammering our endpoint and causing, um, a lot of database queries to happen for their licensing lookups, subscription lookups and this, and there was no caching layer in our code that would be passing those that result back. Um, and then also the, the solution of the bandaid solution by Pagely. Um, but then, um, then also providing insight on who those sites were and eventually us, you know, given the go ahead to start blocking some of them. Um, you know, so we could actually work on a fix and, um, you know, it’s, it’s never fun to have these kinds of issues cause it takes, it just drags away from our core business functionality. But, you know, it’s definitely a, uh, an issue that needs to get resolved because our most important asset is our website. You know, it’s where people go to for support. It’s where we make a lot of our revenue. So, um, there was a recommendation to go to a, um, I believe RDS, dedicated RDS or is that relational database if that’s an AWS term? Right. Um, yeah,

Arman Zakaryan: 00:24:05 A private mySQL server basically.

Devin Walker: 00:24:08 Yeah. So we still have that implemented today and we noticed that immediate speed improvement when that happened. Um, we previously did not have that, I believe before this. And then we also bumped up, I believe to a VPS-2, not during this time, but a little bit after that, uh, to give it a little more CPU power. And then, uh, we recently, you know, got, took care of a lot of these sites that were having issues. So we were able to downgrade back to the VPS-1. But the way that the architecture can scale up and down is really nice. Again, the costs are um, a little more, sometimes I would want to spend for hosting, but we have to because, um, one – we need that reliable site and two – we need time to work on our solution. So, you know, we definitely appreciate all the support. So we’re just getting more than I’m sure, uh, is, is… the value is definitely there is what I’m trying to say.

Sean Tierney: 00:25:09 Yeah for sure. Arman, from your perspective, I was looking through the tickets and I noticed, uh, when you said we’re going to have this work in more of a traffic shaping manner versus a rate limiting manner, can you talk a little bit about what that distinction is and what you mean there?

Arman Zakaryan: 00:25:30 Uh, yeah. So when you have situations with, you know, random things out there hitting your site and you’re trying to slow them down, uh, if you just return, uh, a rate limited message right away, you’re not going to slow them down cause they’re most likely not paying attention to the response they’re getting from the server they’re talking to. And so, uh, one of the things that, that we can do very easily with Ares is we can delay the response on those requests that we, that we know are bad. We basically put them in the slow lane and we, we let the server go back to handling other requests that are valid. And we put this one Kinda to the side, you know, real, we’re responding 10 seconds, 15 seconds instead of a second. And just by doing that, you’re basically regulating how fast that outside entity could hit your, your server.

Arman Zakaryan: 00:26:30 Cause it’s still waiting for a response to come back. So it’s either gonna hit some time out, it has configured for itself or it’s just going to sit there and wait for a response from you instead of making another, which could, you know, it could cause even more problems for you. Uh, if you’re, if you’re blocking the request now you have, uh, even more requests coming in and now you’re consuming more bandwidth and you’re taking up more CPU power from your server in returning a, a invalid or forbidden or rate limited message. Uh, so that’s really the distinction there. If there’s stuff that we know, if we just block it right away, it’s just going to keep coming back at a higher rate than we can figure it to delay the response. Or we shaped the traffic so that we’re, we’re pretty much calling the shots on how that’s gonna go. You know, we’re not, we’re not hoping that that thing making the request is going to listen to the rate limited message and slow down on its own.

Sean Tierney: 00:27:30 Yeah. So let me see if I can, cause I just got at what I think might be a really good analogy here and it, or it could be really weird, but this is what’s in my head right now is I’m imagining if, let’s say that Pagely has that there’s just like a spammer trying to war dial Pagely and he’s calling these numbers and he calls Arman and he says, Hey, is Arman there? Hey, is Sean there? Hey, is Josh there? It’s like if you just hang up on him right away, he’s gonna just keep calling back and call someone else. And so that’s really going to not do anything to help you out in that scenario. But if you say, oh, hey, let me, yeah, Arman’s here, let me go get them. And then you just leave the phone off the hook. You’re now kind of like, that’s spammer is now kind of waiting indefinitely in a holding pattern until you get back to them. And so that it’s, it’s mitigating that, uh, essentially like not DDoSs but it’s mitigating that flood of traffic that’s coming in. Is that a valid way to think about it?

Arman Zakaryan: 00:28:21 Yeah, that works.

Sean Tierney: 00:28:24 Okay. That was just a weird thing that kind of popped into my head. But like it maybe for the less technical people that makes more sense. And that’s certainly how I am.

Arman Zakaryan: 00:28:35 Basically we’re just putting it in the slow lane, you know, using, using the features available in that that layer of things.

Sean Tierney: 00:28:45 Cool. All right, well anything else on the rate limiting stuff? Cause this was like having read all these tickets, it seemed like this was a good chunk of things was figuring out how to calibrate the rate limiting.

Arman Zakaryan: 00:28:58 Yeah. So I mean the, the, the rules we, that we use, you know, that sort of evolved over the past couple of years. Um, I think the first implementations we had were resulting in some false positives who was resulting in legitimate visitors getting blocked. Uh, so we kept refining those. And then I think, I think the rules we have right now, uh, we switched them to Aries a few months ago and the rules we’ve been running, uh, recently, I think, uh, I’ve been pretty good for the most part. Um, you can, you can correct me if I’m wrong about that Devin, but, uh, we’ve been trying to avoid walking, you know, this, it, that user creation or that site, uh, and we’re trying to stick to like more generic rule set that that applies automatically. That doesn’t lead to false positives, hopefully. And you know, anything that’s not doing a crazy rate, it’s not even gonna notice that there’s any rules in play at all.

Arman Zakaryan: 00:30:03 Um, but anything that is going over that, that it will be, um, you know, it’ll just be, but it was slowly like we just said, and it will, it’ll keep it from a vacuum server. Um, but yeah, those, those settings are all really a tune-able on our end. You know, we can, we can configure a bursting threshold. We can configure a maximum rate per second. Uh, that’s something you should be hitting you at. Um, you know, we can do all kinds of things, uh, to, to fine tune that, uh, if it’s, if it’s still not working as expected.

Sean Tierney: 00:30:36 Cool. Yeah. And it sounds like anytime you’re just chasing specific domains or specific user, user agents, it’s just whack-a-mole. Because then it’s just until the next one comes up. Now you’ve just got to block that specific one. And so you really need to get to a more generic approach where you can address it via rule set, which it sounds like we did.

Arman Zakaryan: 00:30:58 Yeah. And I mean, not just for this, but you know, all kinds of stuff. We see all kinds of traffic patterns and we’re always adding new types of rules to Ares default ruleset, uh, where we’re looking for specific patterns that we observe to cause problems for other sites. And we, we sort of distill that down to everybody to benefit from, from all those new rules. Um, and some of those are performance improvements. Uh, some of those are, uh, just correcting bad behavior from, from external sources. Uh, some of those are really good security countermeasures. Uh, so that, that’s the whole sort of, you know, happens for every customer using areas without them really noticing anything on their site changing. Ideally, you know, it doesn’t cause any sort of a fallout, but we’re always adding new rules based on what we’re seeing happening to our, our various customers.

Sean Tierney: 00:31:54 Cool. All right. Um, this next one is a fairly trivial thing. Trivial in the sense that we bump into it a lot, so we have a solution for it, but it looks like easy digital downloads sets a cookie. They have a session start function. And so on the homepage of GiveWP, we were noticing it was bypassing cache. Um, can you talk about what we do there in terms of, I know we’ve got n MU plugin patch that we apply, but what is the solution for that issue?

Arman Zakaryan: 00:32:24 Yeah. Um, so any caching layer they use, whether that’s Varnish or NGINX or whatever, uh, the default behavior is if the responsibility that cookie header line, uh, like set cookie PHP session ID is this or it’s that cookie. Yeah. Some other custom cookie name. Um, so that, that’s, that’s a very clear sign to the caching layer that this content is going to be personalized and that it should not cache it. So if you’re doing something where it’s resulting in your homepage, responding with, with the set cookie and your home is not going to cache, and that’s arguably the most popular page on your website in most cases. So, uh, if the homepage is not, uh, hitting the cache, then your server’s working harder. Uh, eat more CPU, you need more workers. Um, so the, the fixed there is a very simple, uh, wordpress and you plug in that just adds a filter, uh, where if you’re a, the URL is the homepage, then, uh, it just removes that response. It just removes that set cookie from the response before it sends it back out, uh, to NGINX to process, uh, the response to the client. Uh, so with that, with that small tweak, you know, we’ve got the homepage caching again and it’s not causing all this, uh, all these load issues anymore. Cool.

Devin Walker: 00:33:55 Yeah, if I recall that correctly, I think it was more than just the homepage. I think it was, uh, sessions we’re starting on almost.

Arman Zakaryan: 00:34:04 Yeah, we’ll be starting it on every page. But I think, I think the, I think the [inaudible] plugin we added for you just remove it from the homepage. But,

Devin Walker: 00:34:13 Um, yeah, I remember doing some custom code for that too, and ensuring that it’s only started on the necessary pages, like the checkout for sure.

Arman Zakaryan: 00:34:26 So, um, there’s a few different ways to address that kind of problem. Right? Uh, there, there are some posting set ups out there that straight out, straight up ignore that set cookie line in their response and they don’t, they don’t consider that a reason not to cash. Um, and I mean that’s, that’s obviously like the easiest way to do it and not, it doesn’t generate a bunch of, uh, support inquiries. Like, why isn’t my stuff caching is basically a forced cash. Um, but it’s really important to not do that if there is actually legitimate reasons to have to have a session could keep her visitor. Um, so, you know, we have, we have our stuff set up sort of by design. Like, okay, we’re gonna stick to this standard of set cookie and the response means cache miss. It’s going to necessitate an interaction in the conversation with, with our customers about it.

Arman Zakaryan: 00:35:24 And then we can work with them to establish workarounds that would hide that response or address that issue with within the application code so that it doesn’t send that cookie out. Um, but you know, it requires a little more interaction and a little more more touches to, to get to the bottom ofsomething like that. But you end up with a better outcome I think, because as you just indiscriminantly cache any content, then you could end up cashing something that’s personalized and you may be, you may be bobbed and you go load a page and you can see something for Sarah, you know, that’s, that’s not really what you ever want to do.

Devin Walker: 00:36:04 Cool.

Sean Tierney: 00:36:06 All right. So there was just kind of a smattering of various tickets around CDN and SSL and stuff like that. Uh, but I think the next media one was September 10th, uh, related to new relic. And now Devin, do you guys use new relic? Cause I know we use it and diagnosing stuff, but do you actually use that as well?

Devin Walker: 00:36:26 Hmm. Um, we used it on suggestions for monitoring from, uh, from you guys. Um, so that was why we got on it. Um, I’ve used it previous to that, um, over the years. But, um, I believe that’s why this was on her server and it was just a unique thing that new relic does. I believe that the support team pointed out where exactly I can change this option. Who, um, I think it was the browser, um, stats that they were, uh, that was, this was causing and that was just when you, uh, it’s just one of these, again, very weird, um, unique situations with the nature of selling WordPress plugins and, um, and how that’s displayed in the customer’s feeds when they were looking at, uh, one portion of our plugin that pulls content from our site and brings it into the actual panel, sort of like a, an add on directory within our, our GiveWP platform. So, um, yeah. And we didn’t want to display that. Um, and, uh, as soon as I can has it been a problem? But, uh, again, the support team helped, uh, me go find that option and turn it off.

Sean Tierney: 00:37:47 Cool. Yeah. So that, that’s a good example of something that’s, you know, obviously we’re not new relics so it’s, it’s hard for us to support it, but we can certainly be that router that, uh, it can be, cause we do bump into this type of stuff. We can then recognize it and then point you to the right resource for solving it.

Devin Walker: 00:38:02 Yeah. New Relic, incredibly powerful. But there’s a lot of things going on there, so I was appreciating that helped there.

Sean Tierney: 00:38:10 Yeah, for sure. I’m okay. Arman, on this next one, uh, mySQL column size too large, can you talk us through what that is about with the whole my SQL innode DB mismatch?

Arman Zakaryan: 00:38:25 Yeah so we, we use Amazon Aurora, uh, for the most part. Like most of the sites that we go saw occasionally are, are backed by an Amazon or, or a database or, and the, a big difference between Aurora and a regular, my SQL engine or Maria db engine is that it only supports, you know, db as an engine format. Um, if you’ve been using my SQL for awhile, you probably have heard of myISM. And really this is, this is where, um, these kinds of compatibility issues start happening is there are plugins out there, many plugins have their database schemas if they are using any custom tables, uh, inside of the wordpress database, uh, that really been designed historically for my set engine, for the table format. Um, and so in this, you know, in this new world where everyone’s using Amazon Aurora, and, uh, even without Aurora, you’re using, innode DB, uh, cause it has a lot of, a lot of better, uh, features for a per row locking. Uh, it’s just, you know, flood better. Uh, it runs a lot faster. It allows you, uh, to do more things without doing so many locks, things like that. The data is more portable.

Arman Zakaryan: 00:39:52 You have to take into account some of those differences in Schema basically. And so there’s, there’s a couple of things that you may have to change and your database schema before you create it. Um, but there isn’t really, there aren’t really that many situations where a slight tweak to the schema would prevent you from, uh,

Arman Zakaryan: 00:40:14 being able to use, innode DB instead of myISM. Uh, but with that said, because we use Amazon, uh, to host all our sites, uh, and because we offer private RDS instances for people if they want it or if they need it, then we can absolutely support vanilla mySQL RDS or a Maria DB. And if you have some reason that keeps you from actually being able to use innode DB or Aurora, we can still accommodate that. And it’s the same pricing, uh, regardless of what engine type you asked for.

Sean Tierney: 00:40:50 Alright. So with the private RDS, we can accommodate whatever, uh, table engine you need to use or database flavor. But in this scenario, we didn’t actually even do that. We just, uh, there was like a, a coding SQL parameter row format equals dynamic it sounds like.

Arman Zakaryan: 00:41:06 Yeah. That’s part of the Schema. So yeah, when this case they were able to just make a slight tweak to, to the Schema and they were up and running.

Devin Walker: 00:41:15 Cool. Hmm. I’m pretty sure if I remember correctly, this is like a proactive thing, um, that Pagely did. I, I wasn’t even aware of this. I think I was, it was one after the fact and um, yeah, there’s been one or two occurrences where that’s happening and that’s something really cool to…

Devin Walker: 00:41:34 Proactiveness, right? That’s important.

Sean Tierney: 00:41:36 Yeah. I mean, we certainly do. I know what a moderating capacity, we try to head off issues before they manifest as downtime. So in a lot of situations, if we see some resource spiking up, we can actually get ahead of it and before your site ever even goes down, we can at least address the issue. So it sounds like this was one of those cases. Cool. Uh, alright, well, moving on. This next one, it’s on my birthday, May 7th. Um, so purging requests causing a long response times. Uh, it sounds like this was addressed via a [inaudible] plugin. Armand, is anything, any lesson here or anything you want to comment on this one?

Arman Zakaryan: 00:42:17 Uh, yeah, so this, this goes back to the, uh, automatic purge hooks that I was talking about. So whenever you update your pages or different posts, um, there’s a hook that triggers that we’ll go and issue a purge request, uh, to the, to the caching layer. And so, uh, there are cases where sometimes you have a plugin, are you doing something with a custom post type? And, uh, as you’re, as you’re generating through all these items, or you have a cron that’s updating a bunch of things that are of a custom post type, they’re not necessarily, you know, pages on your website. It’s just using a custom post type to to do what it does. Um, and so we, we have, we have the ability to specify which custom post types to skip that purge hook for basically. And so if you have something that’s updating a bunch of things and they’re not really public facing pages, they’re just as composts, um, you really don’t need all those items to get a verge, a request made.

Arman Zakaryan: 00:43:24 And just the overhead of having to scan, uh, all the, all the items in the cache to know what’s purge can add overhead. Uh, so that’s a really simple workaround. We just, we had a, we had a new definition inside of, uh, the section where you excluded, you know, which, which custom post type to exclude from the perch. Um, and then that, that just kind of stops the issue from happening. Of course we log in as well, so you can see what is being skipped. Um, that’s the, that’s a good way to just have that extra sanity is to know like, oh, this is working or it’s not working.

Sean Tierney: 00:44:03 Right. Cool. Um, alright, so this next one, May 15th, uh, it looks like we be in an IP block that was hitting it, the server, uh, creating excessive load. Um, I mean we’ve kind of already addressed this whole, how we do that. Uh, in terms of not just whack them all with specific IP addresses but trying to do a more robust approach to be a rule set. Uh, so I think we already covered that one. Maybe we just skip over that one. Yeah, yeah.

Arman Zakaryan: 00:44:33 Yeah. So I think just the, it’s still important to, to to highlight that event because this, this was before we added the more generic roles. I think at that point they were already on areas but uh, we were, we were maintaining a list of uh, sites or its uh, two to block. Uh, we hadn’t, we hadn’t yet added the more generic rate limiting traffic shaping policies. So I mean it is good to maybe just make mention of it because you know, it is just continue to be a game of whack-a-mole basically until we got the better rules in place.

Sean Tierney: 00:45:14 Here’s a question about those rules. In the way they work. Are we able, let’s say we see the same pattern of behavior occur with all easy digital downloads sites and that we just come to find that there’s a rule set that always makes sense. Are we able to, are those portable? Can we just apply, like move areas, rules amongst customers and actually repurpose those?

Arman Zakaryan: 00:45:36 Yeah. So we do that sometimes. Uh, if there are really, um, generic and safe types of, uh, rules to apply everywhere, then, then we have done that in the past. Uh, we actually do that pretty regularly. Um, but it, it really depends, uh, you know, I think sometimes you do need it to be very custom tailored to, to each specific site. Um, I know, you know, we know how to dial it in for Devin because we’ve gotten a lot of good feedback from him, uh, and has seen a lot of communication, uh, to understand like, okay, what, what are the sites that you’re, that are running your plugin like supposed to be doing? And we’ve learned, okay, there are some valid reasons for it to make six requests in a row, but they’re not valid reasons for it to make six requests in a row every minute.

Sean Tierney: 00:46:26 Yeah.

Arman Zakaryan: 00:46:28 So it’s really, you know, it’s, it is possible to make a sort of generic rules that apply for everybody. Uh, in some cases it still takes a little custom tweaking here and there, uh, to, to, to fit the use case.

Sean Tierney: 00:46:43 Nice. So maybe that as a starting point, like if we identify kind of best practices with WooCommerce or EDD and we can implement kind of a base rule set and then tweak them per customer. It sounds like that’s kind of the happy medium there.

Sean Tierney: 00:46:58 Cool. Um, and Devin, just again, Kudos to you having reviewed all the support history. I agree with our, like, uh, the, just the, the level of back and forth and on parody just, uh, W it’s not always the case. You know, we wish that every customer had your level of patients and, uh, you know, interaction and what not with us, but, uh, it’s a good history of tickets there. Yeah, no, I’ll appreciate it. And, uh, you know, I’ve come to trust

Devin Walker: 00:47:30 Pagely a lot and, um, and then, you know, I sometimes feel bad for the number of tickets that I put in, but, um, I, you know, I repaired it with a good feedbacks and testimonials and, um, and sponsoring PressNomics and stuff like that. So hopefully we’re good to go.

Sean Tierney: 00:47:50 Yeah. Yeah. Honestly, you guys are not even a pain to deal with. We love working with you. We love helping you.

Devin Walker: 00:47:55 It’s obvious problems to have.

Sean Tierney: 00:47:58 All right. Moving on. Uh, July 11th, intermittent five oh three errors. I think this is what we talked about before in terms of providing a bandaid fix through hardware. And even though it’s, we, we don’t like to go here first, you know, that to me, papering over it with a hardware fix is we would far prefer to solve the root issue, which is what we normally do. But in this situation it can be a valid thing. It sounds like you were working on a code fix at this point, so you really just needed to buy some time, uh, with increased hardware and that’s exactly what we did there.

Devin Walker: 00:48:32 Yeah, we’d started the code fix around that same time and we’re now just going to start finally rolling it out to production next week. Um, it’s been tested, uh, for the past week and a half on staging and it’s been going pretty well. So we’re confident that we can bump it over to production next week.

Sean Tierney: 00:48:53 Cool. And what type of like rolling it out. So now when you do that, it’s now suddenly available and people can start downloading it, but there’s no guarantee every one of your customers is going to download it. What type of adoption do you see when you roll out a fix like that? Like what, how soon until it proliferates and people are actually all using it?

Devin Walker: 00:49:12 Well, it’s not really about like them downloading and updating because of the course. Um, I was a lot of these wordpress sites and the nature of wordpress, they’ll be on an old version for the lifetime of the plugin. So, um, what Pagely is done is actually route that traffic through to a new server that we are controlling on RN. Um, and it’s basically our licensing proxy server and they’re just, uh, Pagely has just been, uh, passing that through on the staging and looking for a user agent when it comes back. So there’s kind of this complex loop that happens. Um, I’m not the primary developer on it, so I can just only give you the high levels on it. Um, but that way we can ensure that all of these requests come over over time and we will push out a code update so that the end point, you’re our point to our new licensing server.

Devin Walker: 00:50:08 But again, this will take use to privilege late through our entire ecosystem of users and maybe some will never get it. You know, WordPress is just talking. Now, uh, I don’t know if you saw WP Tavern article about proactively updating old sites to 4.7, and I’m like, yes, this is good. Let’s do it. Um, but then there’s all these other people who are like, don’t touch my code. I’m more in the mindset. Let’s, uh, let’s all try to use the latest version. Um, use a reason why when you open up chrome or some of these other applications that they always keep it up to date. And um, I liked controlling my, my customers experience and environments and it makes life a lot easier for us on the development end point.

Sean Tierney: 00:50:53 Yeah. It certainly, I had a startup years ago called jump box and we dealt with distributing software. We were specifically distributing virtual machines that contained entire stacks of software. And so we intimately know just the pains as soon as it’s no longer SaSS, as soon as you no longer control that environment where you can just update everything and know that it’s all set. Now you’ve got just this, you know, unknown quantity of VM’s out there at varying states and varying, well, you know, levels of security issues. Now it’s- what a nightmare. So

Devin Walker: 00:51:27 Yeah, and not about you, not to drone on about the subject, but like backwards compatibility to, we’ve written a lot of codes, make sure that this works with doctors and this works with this version and the add on model too. You know, if I have a recurring donation plug in on 1.9, that one and the cores at a much lower version, they need to be able to talk properly. So we have to develop all these clever workarounds for just the nature of the business that we’re in.

Sean Tierney: 00:51:58 Cool. Um, all right, well, so we’re, we’re getting close here to the one hour mark. Uh, it looks like we’ve got two more issues to talk through, so I’m going to go to the July 25th cron. Uh, so this, yeah. So well do you want to, I’ll just let you describe what was the issue.

Devin Walker: 00:52:15 Yeah, so we were, you were, we’ve been using Amazon SES for, um, sending emails through our WordpPess sites for a while now. Um, but the plugin we were using, um,

Devin Walker: 00:52:28 has been abandoned for about, you know, two or three years. And then I knew it was on shaky ground. Um, the guys had delicious brains, really great developers have this WP offload SES plugin, which is a much more improved experience and much more reliable plugin. Um, but I couldn’t figure out why the heck it was working on our one WP business reviews site perfectly and not working on give them a VP. And, um, I didn’t, I had no idea. So I was like, all right, let’s, let’s ask Pagely. I usually know how to figure this out. And, uh, we went back and forth on a couple of different items, but they finally came back and said, hey, you know, it’s a curious caching WP Cron, that PHP, which, um, which avoidance yes. Relies on to, to batch and send out these emails. And I’m like, wait, isn’t this a service for WordPress? Like why would they be caching this WP cron father? It doesn’t make any sense. All I mean, but lo and behold, they were, and so I had to go and submit a support request to security, not to talk trash about them or anything, but it took them a couple of them much longer to figure that out. Um, but they finally did and it started working. Um, and now our email system is up and running and reliable again. But you know, security is, um, finicky service at times.

Sean Tierney: 00:53:51 Yeah. Well that seems like something that would be applicable to pretty much everyone. So hopefully that’s a, uh, hopefully they took that scenario and bake that in as a rule for all sites, I would think. Cause I can’t imagine a good reason to cash WP cron.

Devin Walker: 00:54:05 I have no idea why they would either. And I need to ask Tony or Dre or somebody over there see what’s going on.

Sean Tierney: 00:54:13 Cool. All right, well last one here and then we’ll wrap up. So on August 5th, uh, endpoint redirect solution needed for EDD licensing queries, um, I think, can you, can you, I know you said that you’ve got the developer, so you can just talk only at a high level about it, but can you explain what the change was made in terms of using Google app engine and what you guys were doing there?

Devin Walker: 00:54:36 Yeah, so we’re using Google app engine to run a, a lumen based limit is like a lightweight version of Laravel. And what that’s doing is, um, and there’s also a red is caching layer in between. Um, but essentially it’s guide to mitigate the work that our database does on Pagely to look up this, uh, licensing checks, activation checks, um, all the different checks that, uh, the plugin does to ensure your up to date and you’re getting updates that you need. Um, all those are going to be cached on our app engine. And we’re also tying into, um, via wordpress folks with our wordpress site. So if I go in and I released a new version of whatever ad-ons and I click update, that’s going to tell our proxy server, okay, you need to get some more additional data here so that when people check for updates that they’re going to get the proper version.

Devin Walker: 00:55:32 The same thing goes for customer records. If I go onto a John dos customer record and I, but, uh, you know, a different name or whatever that’s going to fire and Hook, update that customer record on the proxy servers so that when a, his site comes and checks that, he’ll have that proper data that comes in there. That’s kind of like the high level solution there. Um, and then again, going back page, really I, we requested that day. Um, you know, they pass through all traffic to that end point to our new server and then when we need to get data back from that end point that they only accept traffic coming from us. So we have a specific user agent that they should be looking for.

Sean Tierney: 00:56:16 Cool. Cool. And I’m on, no, it was all straight forward to add or, yeah, I was just gonna ask you what [inaudible] that was pretty easy to do. Uh,

Arman Zakaryan: 00:56:24 that was just matching on two conditions, uh, this location and whether or not it’s this, uh, authorized user agent and then, um, setting it upstream to be, uh, the customer point that they, that they have running elsewhere. Uh, so again, that’s really easy to do. I mean, you could do that with engine x normally as well, and you just do proxy pass, uh, inside of a location walk. Um, but yeah, that’s

Devin Walker: 00:56:53 so, it’s really cool that visually it helps in there. You know, it gives us that, um, extra level of support cause we tried to do it on our own for, uh, the engine x rules. And remember just saying your cat, this doesn’t work in front of this last page.

Arman Zakaryan: 00:57:09 Well yeah and we run into that same issue sometimes if we’re just dealing with normal engine x is like I’m having trouble getting this rule to work properly sometimes, but that’s why we made areas. Um, and I, I could, I’ll send you a, I’ll send you Devon like a sample of what that rule looks like. If you’re curious how it’s expressed in our area’s format, it might, it might really help help you realize like how much easier it is to do it our way.

Devin Walker: 00:57:38 I think one of the developers that was working on this asked me, can you, can you send me what the rules they did? Cause I’m curious cause I couldn’t get this to work. And then yeah, I believe their response was like, oh, we have our own way of doing it, you know, but we’d still be interested to see how much more. Yeah.

Arman Zakaryan: 00:57:56 Uh, it’s, it’s just a, it’s just a bit of Jason Code. It’s not something that you could really load anywhere else cause it’s, it’s tied to our aeries system. But, uh, but yeah, it’s, it’s a really easy way to, to match on different things that are usually a little more challenging, especially like multiple conditions, this URL and not this user region, um, or this user agent or whatever other parameter. Um, but yeah, uh, it’s really easy for us to set up either an outbound or inbound reverse proxy of anything custom. Uh, we can do normally schticky or we could do gps. Uh, we can set up multiple targets, you know, it is still a balancer, uh, inside of engine x that we can make use of. Um, so yeah, and then if there’s any issues with that, uh, we can, we can easily see like logs on our side or via airlocks and can show if there’s any sort of, uh, issues with the, uh, with the secure connection negotiation. If there’s something wrong with the way the request is being sent and maybe the host header is wrong or something like that. Um, but yeah.

Devin Walker: 00:59:05 Cool. Cool.

Sean Tierney: 00:59:08 Well Phil, as we are right at the one hour mark, so I think that’s probably a good place to wrap it up. I’m Devin, I wanted to give you the opportunity cause we’ve been talking a lot about give WP but you’re involved in some other stuff, the WP business reviews and give Io, uh, I’ll just

Sean Tierney: 00:59:22 give you the opportunity to plug whatever you’d like to plug here.

Devin Walker: 00:59:25 Yeah. Cool. So, um, WP business reviews is, um, a product that allows anybody with reviews on, um, the top, uh, platforms like Yelp, uh, Facebook and Google, including several others to pull those reviews in, um, place them on their website. And also, um, as you get new reviews over time, they’ll also be told, and so there’s some automatic aggregation there as well. You can use that for social proof. You can use that. Um, you know, for testimonials, you can add your own testimonial style reviews in there. So it’s pretty flexible and that’s relatively new product. Um, but again, we’re really focused on solutions for fundraising and nonprofits. Um, and, um, there’s also some integrations with some nonprofits services within that too. Um, but the main thing we’re really working on too outside of GiveWP is, um, a new, uh, solution called [inaudible], which will, um, be, it’s a Laravel based application.

Devin Walker: 01:00:27 Um, also on Google app engine. Um, it’s just now getting sort of, um, ribbon in the five months into development now. So our target is to launch that and um, in a, probably Q three, 20, 20 next year. And so there’s some time there, but really the focus on that is, um, the backbone of it is to, um, get and retain the most recurring donors as you can. So everything about that platform will be built to focus on, um, obtaining and maintaining a large recurring donor base. Um, so yeah, we’re really excited about that. I’ll have a lot more information on that next year, but, um, it’s the big thing we are working on right now.

Sean Tierney: 01:01:13 Very cool. Um, I actually run something called charity make-over on the side, which is just like a volunteer side project thing that I do. Uh, essentially like periodic hackathons, pulling together stuff and implementing different, you know, things for charities like that. Uh, we used ignition deck on the last thing. We built a, it’s a crowdfunding platform. But yeah, I mean if you need Beta testers or anything that we, we’d love to maybe try that out and implement that for someone all pro bono.

Devin Walker: 01:01:40 Awesome. For sure. Yeah, we’re going to open it up, uh, probably, you know, March, April, next year, and really make sure that it’s got solid ground before it’s actually released. So I’ll, uh, I’ll definitely give you a call on that.

Sean Tierney: 01:01:55 Cool. All right guys. Well, any final thoughts, Devin, for someone who’s thinking about using Pagely uh, it sounds like you’ve had an overall positive experience. What would you say to someone who’s considering working with us?

Devin Walker: 01:02:07 Oh, absolutely. Do you have custom situations or you need extra support with the super knowledgeable people? You don’t want to maintain your own servers. Uh, you want to focus on your business, um, which, uh, the core of what you do a move to Pagely. I’ve tried every single, not every single, but almost every managed wordpress host under the sun. Um, and there’s some really good ones for depending on what you want to do. You know, I’ve tried to Kinsta, Siteground, WPEngine, Liquid Web, you name it, I’ve tried it and nothing. Uh, no level of support has been as good as Pagely. Um, not to mention architecture in the server system. Just everything’s good. It’s a little bit more expensive than somebody else, but you know, you get what you paid for.

Sean Tierney: 01:02:54 Cool, man. All right, well thank you so much Arman. Thanks for your time today. Uh, we’ll wrap it up there and fellas have a great day. Thank you. Cheers.