Memcache and Hotaru CMS testing for v.1.6.0

I have added memcache as an option to the core of Hotaru CMS and am looking for a few sites that want to test it. If successful this will be standard in the v.1.6.0 Hotaru CMS version which is planned for release by the end of this month.

Currently I have memcache running on http://ipadrank.com in debug mode and am displaying the stats at the botom of the page. I have added a counter showing the number of cached queries in addition to the db queries.

I have set my cache memory to be only 20 seconds here so that you can see the query numbers changing.

At the moment I have set up memcache to do the following:

When Hotaru CMS goes to read from the db it first checks in memcache for the query.
If the query already exists and is not more than 20 seconds old, then the results data from the query is shown instead of querying the db
If the query does not exist then Hotaru CMS calls the db and saves it to memcache as well as returning the data for the user to see

If this test is successful and people are interested the next step would be :

When inserting a new record, to save the query and data to memcache as well as saving to db. This means that the next time Hotaru CMS looks for the data on a SELECT call to the db it will already be in memcache.
This will also mean that data gets updated in memcache without waiting for the 20 seconds to expire before the db gets a fresh call. This can apply to INSERT, UPDATE, DELETE and any other commands
The advantage of this second step comes if you set the memcache expiry to say 10 minutes or something large. That way the data should always be uptodate without having to make the user refresh a large amount of records


I have modified the mysqli driver version of ezsql to do this and so at the moment it will not work with the mysql driver version.
Note: there are 2 drivers we support for the mysql database. They are called mysql and mysqli. It is confusing to have the same name for driver and database, but if you are unsure turn debug mode on and look in the footer to see what your driver is.

No changes are required to the database to use memcache but you need to have memcache installed on your server. If you are unsure let me know

When we move this into the release for v.1.6.0 we will have to consider how to turn memcache on and off in the admin panel and how to balance it against the current db_cache which is file based. Since queries will be saved to memory with memcache there will be no point in also saving those same queries to disk. When testing you will need to turn off db_cache.

Things seem to be working well on ipadrank with memcache but I want to get an idea of how it works with different servers and whether people actually feel a different in page speed.

Compared to popular CMS systems like Wordpress which typically can require 500 - 1,000 database queries to construct a page, Hotaru CMS only has about 50 - 100 queries. This is quite a small amount for CMS's and already produces a fast page load. I am keen to see whether memcache actually makes a big difference or not.

Who is interested to try it out?

 

thanks
I will try that also

At the moment I am noticing that voting and autoreader are entering duplicate records when memcache is on. I need to look at how I can decide when to clear the cache based on the db being updated. Just waiting for the cache time to expire is obviously not good, but we have to decide how to know when a certain cached query should be flushed from memory and reset.

 

yes, understand and if memcache is not installed on the server then Hotaru CMS will still work of course
We are trying to provide options for performance using various tools. The main thing is that Hotaru CMS is already loading fast and has less queries than most other CMS's

 

Version b testing results so far

I'm currently testing version b and this is what I've seen so far:

• After unzipping all the files on the server and going to the home directory/page (i.e., not in the install directory /install/index.php), I see the expected maintenance mode message but also "Warning: Table 'mysite.hotaru_settings' doesn't exist in /mysite/libs/extensions/ezSQL/mysqli/ez_sql_mysqli.php on line 298." Not a big issue considering it hasn't been installed yet, but it might send the wrong message to less-than-technical newbies who, upon their initial installation, get an error.
• When I start the install and get to step 3, after I enter my admin password and click update, I get the message "Ah! You've triggered a CSRF error. That's only supposed to happen when someone tries hacking into the site..." It appears the admin password does not get updated because, when I finish the installation and login, it tells me "Login failed." When I look at the admin record in the database, it shows 'password' in the password field and 'admin@example.com' in the user_email field (even though I did update the email address during installation). So it looks like the email address didn't take either.
• My Selenium plugin install script failed when installing the Follow and tim_thumb plugins, and I see that they are no longer included as defaults. Not sure if that's intentional or not, so thought I'd mention it.

 

After saving settings on the admin Settings page, I receive a red message "Error saving settings." The settings did not save properly. (I think I remember this error from the previous version, but the settings did save properly even though the error occurred.)

 

After installing the Vote plugin, I went to the settings page to change a setting, but didn't. Instead, I just hit the Save button. Received the error "CSRF error. Please try again." with the following message where the settings should have been: "ERROR : [website_path]/content/admin_themes/admin_default/plugin_settings.php not included." The settings were not there.

 

Being logged out automatically and can't get back in

OK, something very weird is going on with logins. I've now emptied the database and reinstalled 8 times, and the same thing happened each time. I manually change the password in the database, install the base plugins, and then might leave it alone for a few minutes to do something else. It appears to be logging me out automatically. But when I try to log back in, I get a "CSRF error. Please try again." Then I reinstall and it works until I leave it alone again for a few minutes. And it's not the browser cache as I've tried in multiple browsers and also shut them down/cleared cache.

Would you mind terribly if I didn't test this version further until this login problem is fixed? I'm just not able to get very far with testing given this issue. As I've never seen a CSRF error on any other website (in fact, I didn't even know what CSRF was until I started playing with Hotaru), is it too early to vote for the elimination of all CSRF checking? (grins)

 

It is strange that CSRF is giving you so many troubles. It wasnt changed that much in 1.6.0 and user signin wasnt changed at all.
This is the reason for using CSRF
https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)

Without it, any forms on your site are vulnerable to someone pretending to be someone else and accessing info

I will run some more tests on login/register etc and see what is coming up, but so far I havent had 1 CSRF error.
Are you operating in a subdomain or subfolder?

 

I am operating in a a subfolder (3 levels down). I spent quite a bit of time trying to troubleshoot what was causing the problem. Even if I delete the tokens in the database and cookies in the browser, I still can't get back in. It's like someone's trying to send me a message to go to sleep already

But I did a complete reinstall of 1.5.2 and the problem is not there, so it's a result of a recent change.

 

Doing a code compare between this beta and the last release, it looks like you changed $h->comment->postId to $h->post->id in plugins/comments/templates/comment_form.php. However, it doesn't appear you changed the one on line 55. Didn't know if this was intentional or an oversight, so thought I'd mention it.

 

Thanks
I will take a look

 

One hit to the database to replace three

As I was looking through the code, I happened to notice the following in /plugins/categories/categories.php:

Code:

        if ($h->cage->get->keyExists('category'))
        { 
            $category = $h->cage->get->noTags('category');
            if (is_numeric($category)) {
                // category is numeric
                $h->vars['category_id'] = $category;
                $h->vars['category_name'] = $h->getCatName($category);
                $h->vars['category_safe_name'] = $h->getCatSafeName($category);
				$h->vars['category_parent'] = $h->getCatParent($category);
            } else {
                // category should be a safe name
                $h->vars['category_id'] = $h->getCatId($category);
                $h->vars['category_name'] = $h->getCatName($h->vars['category_id']);
                $h->vars['category_safe_name'] = $h->getCatSafeName($h->vars['category_id']);
				$h->vars['category_parent'] = $h->getCatParent($h->vars['category_id']);
            }
            $h->pageTitle = $h->vars['category_name'];
            if (!$h->pageName) { $h->pageName = 'popular'; }
            if ($h->pageName == $h->home) { $h->pageTitle .=  '[delimiter]' . SITE_NAME; }
            $h->subPage = 'category';
            $h->pageType = 'list';
        }

Because it appears you're looking to enhance speed in future versions, there are probably some things that can be done with core to remove hitting the database multiple times for the same record. This code effectively hits the database three times for the same row. IMO, if you're looking to increase speed, it would be better to have one function that takes in the name of the column or columns you want to grab and returns it as an array. The way it is now makes it a tad easier to code and read, but it's not very efficient from a database perspective.

Many of you know the code base much better than I, but I would bet there are other opportunities like this to be looking for.

 

thanks. I am going to take a look at this now. Whilst Hotaru is fast and has good caching techniques, there is always room to make things better. Thanks for pointing out this one

 

I have taken a good look at this and rewritten it all, allowing use of memcache if available also. The end result is it reads like it should run faster but I cant see any difference in speed at all.

I also cant see a speed difference when I turn on and off memcache for it

Keep looking through the code. If you see other areas I can take a look. This one was a whole days work to rewrite. It was not entirely wasted but I am disappointed I didnt see any speed improvements from it. I think mySql data calls are faster than many people think

 

Thanx for that effort. My experience with performance tuning has been similar to my experience with trying to learn golf: when I hit it straight, I love the game; when I don't, I never want to play that silly game again. And, yet, it seems beyond me why, when I think I'm hitting in proper form, I shank it

I remember one project where I had 2 performance tuning experts working to fix multiple performance problems for many weeks. Code was rebuilt, indexes were redesigned/added/removed, and nothing that seemed like it would make a difference made a difference. Users could hardly test and these guys just couldn't figure out why their changes weren't having any impact. Finally, one day they did something else and then it all came together. Speed was blazing. Point is: it's not wasted effort - there's probably some other key levers that, once pressed, will make these changes much more effective.

 

This won't provide any speed improvement, as far as I can tell, but I found it curious that the function friendlyToStandardUrl occurs both in PageHandling.php and Paginator.php. It's not entirely clear to me yet why the two functions that seem to do almost the same thing with much of the same code, but thought I'd mention it.

 

I'm not sure if this will provide a speed improvement or not, and this is related to my pursuit of same-named child categories, but I still remain a tad perplexed related to why Hotaru is cycling through the same function multiple times when it already has the answer. Here's how to reproduce the problem: in /plugins/categories/categories.php, add print "query string = " . var_dump($query_string); to line 133 so the code looks like this:

Code:

    public function pagehandling_getpagename($h)
    {
        // Can't get keys from the url with Inspekt, so must get the whole query string instead.
        $query_string = $h->cage->server->sanitizeTags('QUERY_STRING');
print "query string = " . var_dump($query_string);
        // no query string? exit...
        if (!$query_string) { return false; }

Now go to any category and view the source. You should see something like this:

PHP:

string(10) "category=politics"
query string = string(10) "category=politics"
query string = string(10) "category=politics"
query string = string(10) "category=politics"

That suggests to me that, in base and without any modifications, a perfectly formed URL is going through the same function at least 4 times. Is that the way it's supposed to work? If so, would you mind helping me understand why? Wouldn't it be faster (even if only a little bit) for it to get the URL once?