Roadmap Version 1.7

Is there any problem with them at the moment?

 

Can you explain a little more and I will try to help

Cron is already part of the zip download for Hotaru CMS core files
Which is the profile image plugin?
Is this the domain plugin you mean? http://forums.hotarucms.org/threads/domain-results-0-1.1329

 

I think the profile image plugin would require some work to make plug and play but would be a good idea

I will look at a few things but maybe you might have to mention this again a bit later. Perhaps around version 1.8
I am planning to make a few of the admin manager plugins part of the core first

 

The main item for Version 1.7 at the moment is going to be completing the admin dashboard so you can automatically get the updated version numbers of plugins coming through, also maybe the search function, and the same for themes as well.

I am still exploring having some of the admin manager plugins being part of core but that is not quite ready for 1.7 yet. Some of the code and UI in these areas may get an update as well

I havent had any problem with the new mail code but will recheck before deploying 1.7

 

I have been looking at functionality for 1.7 more and think we need to introduce a new ORM as part of the framework. I have looked at this before and tried a few frameworks with Active Record. Each time I found they required a higher version of PHP than people were using. Now that most people are on PHP 5.3 and above (5.6.2 is out and 5.7 is not far away), I think this is worth revisiting.

Some of the basic areas for speeding up sites is in the producing of the content. Yes, we can talk about caching and image processing etc etc, but if we are making too many queries or the queries are too resource consuming on the db then there will be a bottleneck. By using a powerful ORM solution we can rewrite the queries to use less of them, make them more db friendly, and at the same time we can improve the security by having a strong PDO solution rather than the old drivers. I wont explain the difference between mysql, mysqli and pdo drivers because the naming conventions are too confusing. We will still be using mysql as a database, but the driver should be PDO. With a new ORM we may also be able to use Hotaru CMS on postgress db systems as well.

I have been testing some different framework components again recently and have a candidate in mind. I will be making this a big part of version 1.7, so as far as the roadmap goes, version 1.7 will mainly be devoted to a new ORM backend.

In addition I made a number of updates to the Admin Dashboard from 1.6.x so this may get polished up a bit for final release in 1.7

I dont have a firm release date but would like to get it out within a 3-4 week cycle. I may put out some beta tests for this. Once I see the ORM in action I will decide the best way forward.

 

The two areas that significantly improved speed for me on my (now highly customized) build were redoing many SQL queries so that they included joins and stopping the constant churn hotaru seems to go through as it evaluates each plugin. I don't remember the numbers, but it led me to go from something like 500-600 queries sometimes to <50 and blazingly faster page load times. Sadly, it was not easy to do this so there aren't a few quick fixes to make this happen.

The great thing about plugins is how easy it is to add or remove functionality so you can easily customize your build. But this is a double-edged sword and comes with massive inefficiencies. At its core, I think hotaru needs a much better way to parse the URL, determine what data needs to be gathered from the relevant tables based on that URL, and then only load those relevant plugins and make those relevant (joined) queries. But, to do this, I think hotaru would require almost a complete greenfield re-write. For the most part, adding most of the plugin functionality into core will not cause a performance hit if hotaru is smart enough to determine upfront what (joined) queries are required and what code need not be executed. If plugins at least had the capability to check which queries were already slated to run (or had already run and the data available), it wouldn't need to hit the same tables multiple times.

 

Quite a lot of work going into looking for performance bottlenecks in this next release. I am building in Model classes to support the db CRUD operations and when doing that taking a look at how many times the same or similar queries are getting called. Where possible I am bunching these together and/or joining tables to reduce the calls to the db for a single page.

I am also reviewing the file cache operation, which is already very good. Hotaru operates with a number of built in cache areas. Obviously memory is one which is utilised within a page life-cycle, but file caching is also very fast and operates in a number of places in Hotaru. db query caching benefits greatly from this.

So far, the biggest bottleneck which has seen significant improvement in the upcoming v.1.7 is the call to check for pluginhooks. This occurs many times on a single page as each plugin has to find its hooks within the page. One plugin may have many hooks and so the checks for this can take up system resources. In addition to reducing the db calls here (which were mainly being cached anyway), I have reduced the number of times a key function needs to be run which has reduced the page load time.

Since this is more than just a cosmetic change, I will be releasing a test version of version 1.7 just prior to official release. I will release it in the forums under the News and Announcements section so watch out for it. If everything looks good with the first few tests, I will push out an official version for all.

Can a few of you tell me what the load time is for your index page at the moment ?

Here are 3 screenshots from the footer of my index page
For those not familiar, turn on debug mode in Admin and this footer will appear automatically. You can turn it off again afterwards





0.089, 0.096, 0.088

Even though it says "memcache enabled" I had it turned off for this test.

I would really be interested to see what other peoples results are, so we can improve things
thanks

 

Not likely a useful comparison for you considering my v 1.5.1 is highly customized:
Database queries: 50 | Page load time: 0.70s | Memory usage: 15.85 MB | PHP v.5.4.32 | MySQL v.5.5.40-36.1-log

 

ok, the more info the better even if it is customised or a different version. I am interested to see what is considered acceptable to everyone.

I currently have my index page at

Database queries: 38 | Page load time: 0.094s | Memory usage: 9.1 MB | PHP v.5.5.17

Of course it will depend a lot on what plugins are on the page.
I can see on your page you have quite a lot of posts showing on the first page. Is that about 50 posts?

I am guessing that your pagination is handling bringing in all of the posts data as 1 query and then going back to get the comment count and voting count ? or are you making a join for the counts in the 1 query.

I have tried playing with a few different combinations. The easiest was to left join the category and username on top of the posts. The harder decision was what to do with votes and comments count. This Xenforo db actually saves some duplicate redundant data into the same tables for quick access, but I am not keen on that. Another idea is to make views although it seems more SQL'ish than mySQL (albiet mySQL does views very well now).

Maybe a view sucking in all the post, author, category, tags and vote,comment counts for posts on the main list page would be a good performance enhancer ?

I am working on a new Paginatation query and a new filter query for the bookmarking side, so maybe it will come together at the same time.

Thanks for your data

 

One of the places I customized was pagination. I no longer use hotaru's standard; I replaced it with infinite scroll. Yes, the first 50 posts are pulled, but the way they are pulled are much different than hotaru's standard way (e.g., I've combined reddit's front page algorithm with a user-defined algorithm that gives different weights based on category, number of views, number of comments, etc., run checks based on what permissions a user has [category mods can restrict who can view/add posts], and then also throw in a few random posts). One of the things I noticed is that, when doing pagination, you're effectively hitting the database again with the same query but between two limits. To streamline this, I pull the query results once and store them. Then, when the visitor wants to see more and scrolls past the 50, it just does one database query to get the next 50. So, for example, whereas hotaru would normally hit me with 100 queries for 100 posts (i.e., 2 page loads @ 50 each), now 2 page loads requires only 51 total queries (3 page loads requires 52, etc.). Additionally, other plugins can access this data that's already been pulled (e.g., activity, posts other visitors are currently looking at and, for users who are logged in, I've created a plugin for recommended posts using collaborative filtering techniques). (Note: before someone asks, the recommendation plugin is based on my highly customized code, so it wouldn't work for any of you. Otherwise, I'd post it.)

I've made so many changes to comments that I'm not sure I can direct you properly to everything I did to improve efficiencies. But, when looking at an individual post, I'm pretty sure I remember grabbing all comment data (recursively) for that post once (with all associated joins as necessary). I think I remember standard hotaru does a query to get a particular parent comment, and then does another query to get children. Doing it all once saves quite a few hits (although, of course, it increases memory required). Having said that, if you're referring to just the number of comments on the main bookmarking_list page, I grab the comment count when I process the initial bookmarking query (mentioned above) via a join (or a subquery - I can't remember). Then I test if I already have the comment count (from another query) for a particular post_id. If I do, I don't need to go and get it again.

I've completed changed the way categories work in my build. You can now have subcategories within categories that have the same names. The URL is parsed once to know exactly what needs to load. It's a bit convoluted and inefficient what I had to do (because hotaru kept parsing the URL many times/cycling the same function even though it already should have had the information it needed to know what to do), but it works and saves cycles.

Candidly, I've never used views before, but many times I almost convinced myself to learn how they work. It's probably much more efficient than putting query results in a database. (I have a cron setup which deletes them after a certain amount of time so the table doesn't grow huge.) I just never got that far considering I was happy enough with the performance I was seeing. But I think you're right - a view is likely the better way to do it. I just don't know how to use them

 

One other small thing I noticed is that many plugins liberally use

PHP:

        $h->vars['gravatar_settings'] = $h->getSerializedSettings('gravatar');


Particularly, it is getting used some hook areas which in turn may be a loop, like the list of posts on the top page, or a widget showing a list of users. Each time the gravatar is being shown, the settings were being called. They werent coming from the db as there was some caching, but they were getting called each time which seemed a waste of resources. I caught the same thing for who_voted, comments, follow and a few more.

I have changed this on a few plugins and will publish them soon in the Resource section with updated version numbers

Basically, I am putting the settings up here

PHP:

    public function admin_theme_index_top($h)
    {
        $h->vars['gravatar_settings'] = $h->getSerializedSettings('gravatar');
    }


and then using the $h->vars[] call to refer to them in the plugin, probably where the hook is linking in

Now, having looked at this a bit, I was wondering whether it might be better to get all settings for all plugins at the start of each page and keep them in memory, awaiting their being called if necessary.

There was a small divide in the code somewhere which sorted out admin plugins from plain ones to avoid unnecessary calling of anything to do with admin. I might have to look at that again and maybe even give each plugin a type, which could say "admin", or be left blank for others, or maybe an "isAdmin" label. Either way, it would be useful to easily separate out the 2 types of plugins.

As well as reworking the Models for calling the db these type of areas are good ones to improve the performance.

 

There are certainly many things like this that can continue to tweak better performance. From a broader perspective, I think you could probably look at these ideas from a page type perspective. For example, anytime we look at a post page, all post data for that post_id should be in memory, whether that be the content, author, comments, user, etc., but also all data associated with those tables (e.g., all comments and the user data associated with each user, such as gravatar). The /libs/Post.php attempts some of this, but it isn't complete and it doesn't account for table changes (e.g., hotaru_posts) that might be made by plugins. Perhaps it might make sense for get functions (e.g., getPost) to have an argument that identifies what should be gotten (e.g., all data/*, only certain fields, fields joined with other tables). I think this was partially the idea in the design of /libs/Database.php, but it doesn't do a very good job. (No offense intended, of course.).

Ideally, I would suspect the general flow should go something like this:

1. Determine what page the user wants.
2. Determine which plugins are required to run for that page.
3. Determine what data, in total, that page requires.
4. Build the queries based on the total data required.
5. Display the page via the various plugins manipulating whatever data they each need (which is now in memory).

In other words, my suggestion is that, generally speaking, plugins should not be individually querying the database. Plugin developers aren't always taking into account database efficiencies, which is one of the reasons you have performance hits. The plugins should be grabbing data already available to them from core. That's obviously very different than the way hotaru's designed.

 

Definitely some good ideas here. On the single post page, I have something similar to this worked up at the moment, using new Model files rather than the old lib Database you mentioned.

I have some code in the plugins which links up with it, but it is specific to this single post page at the moment. I am working on something a bit more generic which will make it easier to write/maintain a plugin and also help with performance.

A basic Model class currently looks like this

PHP:

<?php

namespace HotaruModels;

class Plugin extends BaseModel
{
        protected $table =  'plugins';

        # explicit pk since our pk is not "id" 
        protected $primaryKey = 'plugin_id';
       
        public static function getAllActiveNames()
        {
            $model = self::where('plugin_enabled', 1)->remember(30, 'plugin_names')->orderBy('plugin_name')->get(array('plugin_name', 'plugin_folder'));
            return $model;
        }
       
        public static function getAllDetails()
        {
            $model = self::remember(30, 'plugin_details')->orderBy('plugin_order')->get();
            return $model;
        }
       
        public static function getFirstFolderForClass($class)
        {
            $model = self::where('plugin_class', $class)->remember(30, 'plugin_folder')->orderBy('plugin_order')->first(array('plugin_folder'));
            return $model;
        }
       
        public static function countEnabled()
        {
            $model = self::where('plugin_enabled', 1)->remember(5, 'plugin_enabled_count')->count();
            return $model;
        }
       
        public static function countByPluginFolder($folder)
        {
            $model = self::where('plugin_folder', $folder)->remember(5, 'plugin_folder_count')->count();
            return $model;
        }
       
        public static function countByPluginType($type)
        {
            $model = self::where('plugin_type', $type)->remember(5, 'plugin_type_count')->count();
            return $model;
        }
                 
}

?>


I am still debating whether the plugins can call this directly or whether they need to step through a $h->getAllPluginDetails() type coding structure

I also havent got a good solution yet for the list posts page but I do like your infinity scroll working and will take on board some of the ideas you mentioned. As I build a new paginator things will come together there. So far most of the changes are working smoothly.

Appreciate your comments. The backend is quite involved so always good to get others opinions