1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Pligg Importer 0.8

Discussion in 'Deprecated Plugins' started by Nick, Aug 16, 2009.

Thread Status:
Not open for further replies.
  1. angolanmade

    angolanmade Well-Known Member

    I didn't think anything changed, the interesting part is that i was able to upload these same files before on a live hosting server, another strange item is that im able to upload all the other xml files, they were all exported at the in the same format with no compression. (categories, comment, votes, etc)
    Is there anything that i can look for in the file that may be causing that error?
     
  2. Nick

    Nick Well-Known Member

    No, not in the file. It will almost certainly be a setting in php.ini. Check the file sizes. I bet the swcms_links.xml file is the biggest, so it's probably dying because it exceeds the limits set by your host. I'm afraid you'll have to check with them.
     
    angolanmade likes this.
  3. angolanmade

    angolanmade Well-Known Member

    Played around with the php.ini and got the import to work again. Thank you!
     
  4. carlo75

    carlo75 Design & Development

    Hi Nick,
    I've some problem in last trunk hotaru rev. 1241 in local, when I try to import Pligg DB. I made all XML valid files, and try to launch pligg importer: categories.xml, are correctly uploaded and imported, links.xml is correctly uploaded, but when I click on Continue (importing) some problems happens:

    localhost/hotaru/admin_index.php?page=plugin_settings&plugin=pligg_importer&file_name=links.xml&step=2

    The page stop loading almost immediately, and in attachement you can see the render of browser and code HTML.

    Where I'm wrong?
     

    Attached Files:

  5. Nick

    Nick Well-Known Member

    Hi Carlo, do you have debug mode enabled? Is there anything in error_log.txt (Admin -> Maintenance ->Debug)?

    Usually, that dead page problem happens when you reach a PHP limit. For example, after uploading an xml file, the whole file is converted into an object here (pligg_importer.php line 250):

    PHP:
            $uploads_folder PLUGINS "pligg_importer/uploads/";
            
    $xml simplexml_load_file($uploads_folder $file_name);
    That $xml object will probably be really big if it contains the whole links.xml file. What's your "memory_limit" in php.ini? Make sure you increase other settings talked about in this thread, too.
     
  6. carlo75

    carlo75 Design & Development

    Ok I forgive to activate debug mode.
    there is an invalid char in links.xml

    C:/wamp/www/hotaru111/content/plugins/pligg_importer/uploads/links.xml:161293: parser error : Char 0x0 out of allowed range in C:\wamp\www\hotaru111\content\plugins\pligg_importer\pligg_importer.php on line 250

    I try to fix it via Notepad++ and then all works like a charme!
    In attachement there is the NUL char screen.
    Thanks again Nick
     

    Attached Files:

    Nick likes this.
  7. carlo75

    carlo75 Design & Development

    Nick, I've encountered another problem.
    I try to import a db utf8_unicode_ci in local installation. Categories, Links, Comments and Tags, after 2 hours of hard work, are succesfully imported, but when I try to import Users (~6.000) I remain stuck on this step process... no errors or messages appears, simply loop running after upload users.xml, and all site is stucked! I need to restart WAMP.

    I check hotaru_users table, and I can see that the import was succesfully (think so), but I can't proceed on the next step, to conclude the entire process. I've try to perform another 2 import, on different PC and I've always the same problem, and everytime I spent 2hrs but I can't end.

    I've check users.xml file and all seems to be correct, I can't figure out where is the problem and how can I resolve. Have you some ideas on this issue?

    How can I bypass the first 5 steps, and import directly votes.xml to conclude importing, without restart the entire process of importing everytime?
     

    Attached Files:

  8. carlo75

    carlo75 Design & Development

    I try this way:

    put under /content/plugins/pligg_importer/uploads/votes.xml

    then in browser type this:

    localhost/hotaru/admin_index.php?page=plugin_settings&plugin=pligg_importer&file_name=votes.xml&step=6

    I see that hotaru_votes table is now growing, but I don't know if the process can be valid or I have altered something. I'm wait message of step end... cross fingers
     
  9. Nick

    Nick Well-Known Member

    Hi Carlo, 6,000 users is a lot of users! For each user, Hotaru runs about 6 database queries as well as unserializing and serializing arrays. It probably pushes your CPU usage up to 100%.

    Is it a Pligg database or SWCMS database? I don't know if this plugin even works for a Pligg 1.0 site. Maybe run the SWCMS DB Cleanup module (in SWCMS) to get rid of all the spammers. That should shrink the database a lot.

    Going directly to a step via the address bar will only work if all previous steps have been completed and the pliggimp_temp table is still in your database.
     
  10. carlo75

    carlo75 Design & Development

    Hi Nick,
    the db is based on Pligg 0.9.5 and I've epurate it from dirt, now I'm still importing votes.xml. Pliggimp_temp table is still in my database, and I completed successfully first 4 steps, I don't know if 5th step (users) can be considerate completely succesfully (I see all users in hotaru_users table).

    However now the importer is still running, when end, I'll give you a feedback on the results. If something goes wrong, I'll retry with DB Cleanup (if is it suitable also for my pligg db).
     
  11. carlo75

    carlo75 Design & Development

    End. hotaru_pliggimp_temp gone.
    What do you think Nick? Importing success?
     

    Attached Files:

  12. Nick

    Nick Well-Known Member

    Well, the temp table gets deleted automatically at the end, so no problems there. I guess you'll just have to test the results. Are the comments on the right threads? Do the posts have the right authors? etc. I hope so!
     
  13. carlo75

    carlo75 Design & Development

    Ouch! Doh! Gulp!
    Comments are ok, but some posts doesn't have the authors!
    This happens because I've killed in advance users.xml importing process?

    How many hours can take the process for importing users.xml, can you estimate?
     
  14. Nick

    Nick Well-Known Member

    I wish I knew! Probably a couple of hours at least. :eek:
     
  15. carlo75

    carlo75 Design & Development

    Just for gossip: OK - Importing process successfully ended.
    Users.xml is really a killer app for CPU and for me! It take 1 hours, but now all seems to work perfectly. I have yet some problems with accented characters, but I hope to solve soon, playing with MySQL collapse and hotaru config file.
     
  16. Nick

    Nick Well-Known Member

    Excellent! Glad you had a successful import. It's hard work, I know! When I was making the plugin, I had to wait ages every time I needed to test something.

    It's not the perfect solution, but the Pligg Importer comes with a "Character Cleaner" (see bottom of the screenshot in the first post). You can change what characters it cleans in /functions/funcs.strings.php in the strip_foreign_characters function around line 566.

    Oh, and don't forget to BACK UP YOUR HOTARU DATABASE after using the Pligg Importer, because you really don't want to have to import it again! :p
     
  17. cybertooth

    cybertooth New Member

    hello ,


    I have about 20,000 + members and huge pligg content ...

    Im using webmin and I have no export utility to xml format only CSv ..

    is there any other way to import db into hotaru?

    thanks
     
  18. Nick

    Nick Well-Known Member

    Hi cybertooth, that's a big site. I'm afraid this importer in its current version (0.8) will choke, shake, roll over and die if it tried to import 20,000 members.

    This plugin reads whole tables into memory, manipulates the data and inserts it into a Hotaru database. For a site as large as yours, you would need a plugin that imports your data directly into a Hotaru database, and then manipulates the data in batches. This would be custom work, so if you're interested, you can PM me or post in the Requests for Paid Services forum.
     
  19. mabujo

    mabujo Designer

    Hi Nick,

    I'm having some trouble with running an import with this plugin.
    I'm running latest 1.2 hotaru on a local LAMP install and have changed the php.ini to have large values for execution times, input times, upload size, memory limit, file size in pligg_importer.php e.t.c. (and have restarted apache and checked with phpinfo that the limits are set high).

    I've run the swcms cleaner before exporting and am left with 38,387 posts and 600 users.
    When I run the pligg importer, it goes through the first categories step but when I come to the link import step and click upload it just returns me to the front 'Welcome to the Pligg Importer' page after a few seconds.
    I noticed that in the category import stage it says Adding.... | but does not give the name of any categories.
    The hotaru categories table ends up with only category with ID 1 and empty values for category name and safe name.

    In the old db there were only 2 categories, the default and one where all the links are related to.

    Any thoughts?
    Thanks.
     
  20. mabujo

    mabujo Designer

    I can now get the files to upload (set really, really high limits - timeouts in the hundreds of thousands, memory to a gigabyte e.t.c.) but nothing is imported.
    I still have the aforementioned problem with categories and all other steps add 0 rows. The users step wiped out the user table completely...
     
Thread Status:
Not open for further replies.

Share This Page