Mike Chaney's Tech Corner
September 20, 2020, 10:37:50 PM *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News: Feb 2013: Qimage Ultimate Challenges... have fun and explore features!
 
   Home   Help Search Login Register  

Download and develop photos from your flash cards with one click!
Get a trial of
FlashPipe today and stop fumbling with explorer windows to transfer photos and videos
Pages: 1 2 [3]
  Print  
Author Topic: Severe performance loss somewhere between 2014.137 and 2014.210  (Read 18293 times)
steveren
Newbie
*
Posts: 24


View Profile Email
« Reply #30 on: May 03, 2014, 06:36:46 PM »

until you can post actual numbers

I have done exactly that. 8 seconds vs 1 or less. These are manually timed so a bit difficult to be more precise, but precision is hardly needed with that level of difference.

I've told you exactly what I did, I've told you what Qimage is doing from the Windows internal POV, I've offered you a VM copy of my test system (ok, it's a bit big to post here!). Tell me what more you want and if I can I'll do it.
Logged
admin
Administrator
Forum Superhero
*****
Posts: 3132



View Profile Email
« Reply #31 on: May 03, 2014, 08:48:51 PM »

until you can post actual numbers

I have done exactly that. 8 seconds vs 1 or less. These are manually timed so a bit difficult to be more precise, but precision is hardly needed with that level of difference.

I've told you exactly what I did, I've told you what Qimage is doing from the Windows internal POV, I've offered you a VM copy of my test system (ok, it's a bit big to post here!). Tell me what more you want and if I can I'll do it.

Well, Terry did a lot more thorough timing and didn't just focus on one operation but rather than you repeating his tests, is there anything you can think of that might cause your network to be slower than normal?  Every single operation you try with images on a local drive is faster in 216 by a significant margin.  And when I test my own network, the margins are smaller but 216 is still as fast or a little faster.  So the only thing I can think of is something out of the ordinary like maybe you are connected to a wireless computer that has a low signal and packet loss, or there is something causing your network to have more overhead than usual.

I do have one more thing I'm going to do which should speed up network access by removing some file existence/date checks but without knowing why your network is acting differently than mine, I'm not sure how much impact that will have.

Mike
Logged
steveren
Newbie
*
Posts: 24


View Profile Email
« Reply #32 on: May 03, 2014, 10:11:54 PM »

It's a GbE network and the remote disk is an SSD. File transfers can happily max out the link at a solid 100+MB/s.

ISTM it's the round-trip latency that kills performance. Are you telling me that in your tests you don't see a factor of 4.5 increase in I/O ops from 137 to 216?

Quote
Well, Terry did a lot more thorough timing and didn't just focus on one operation

But the operation I'm focussing on is the one with problems, not the many without! I'm not for a moment saying that there have been no gains in performance generally, merely that there have been significant losses in an area that affects me directly. Those losses have been reduced in the progression from 213 to 216 but not eliminated.
Logged
admin
Administrator
Forum Superhero
*****
Posts: 3132



View Profile Email
« Reply #33 on: May 04, 2014, 12:23:51 AM »

OK.  Thanks.  Well, I always say... if one person has a problem, however uncommon, there will be more.  So I'll keep looking for more ways to squeeze out more performance.  Smiley

Mike
Logged
steveren
Newbie
*
Posts: 24


View Profile Email
« Reply #34 on: May 04, 2014, 12:49:27 AM »

Thanks for that.

True or not, I've had the impression so far that you don't think my concerns are legitimate. Now I fully appreciate that there can be issues that don't affect most people and therefore aren't top of the priority list, but that's a very different thing to suggesting that the issues don't exist.

Let me just reiterate: I'm a software developer myself and I know full well that customers can have problems that are near-impossible to replicate. Lordy, do I know that! And sometimes it's true that they're complete idiots and everything they say is at best irrelevant and at worst a blatant lie!

I'm trying very hard to convince you that I'm not in that category. I've written file system drivers for Windows - indeed, entire file systems for proprietary OSs - and when I say that a large number of redundant operations are happening, it's not because I'm stupid and wrong.

Now sure, there are times when it's quicker to do something unnecessarily than work out whether it's necessary or not: BTDTGTTS! But I'm sorry to have to say this, but I've had the strong impression that I've almost been called a liar rather than being told that unfortunately my use case has had to suffer for the greater good.
Logged
Terry-M
The Honourable Metric Mann
Forum Superhero
*****
Posts: 3196



View Profile WWW
« Reply #35 on: May 04, 2014, 07:19:33 AM »

Quote
Well, Terry did a lot more thorough timing and didn't just focus on one operation
Some final extra timings I did with 2014-216
I reduced the folder of 299 jpegs to 200, close to the 189 that steveren was using.
Loading the 200 to an A3 page with a size of 6x4" and Optimal spaced was almost instant, 1 second at the most.
I did the same with raw images by adding successive folders to the queue until there were 300. Again, each successive addition of a batch (~ raw 60 images) was almost instantaneous.
It was suggested to do this test:
Quote
Select freehand placement and add an image to a new page at the end of the queue. Edit that page and type a new value into one of the location boxes to move the image on the page
Again, with v216, the move was done in a flash, too short to time.
The results speak for themselves: v216 is OK  Cool
Terry
Logged
steveren
Newbie
*
Posts: 24


View Profile Email
« Reply #36 on: May 04, 2014, 10:20:55 AM »

The results speak for themselves: v216 is OK  Cool

It is not ok for me, and if it's ok for you then there is clearly something different about what you and I are doing and/or our test environments. I would like to work out what that is, not just hypnotise myself into believing my problems are imaginary.

"It doesn't do that here" is an experience that every professional software developer knows well. It does not mean there is no problem.

Let me spell out some more details: I have a custom 295x295mm page, print-to-file set, and I'm placing images freehand. The intent is to create pages for uploading to an online print service to make a photobook. Pages generally have around 5-6 images, just less than 6x4 (147.5x98.4 mm) or submultiples. The images are JPEGs with a resolution that works out to 600dpi at the print size. They are held on an SSD on another machine connected via a Gbe network.

If you can duplicate this configuration and still get instant response, I'd be really interested to see your procmon log. There is no doubt that in my tests, Qimage is doing a lot of redundant I/Os. It may be that this is a red herring: they're not the cause of the slowdown. It may be that it doesn't do them if some other condition applies. Whatever it is, I'd like to know!
Logged
Terry-M
The Honourable Metric Mann
Forum Superhero
*****
Posts: 3196



View Profile WWW
« Reply #37 on: May 04, 2014, 12:31:39 PM »

Quote
They are held on an SSD on another machine connected via a Gbe network
Mike has already given comments and advice on using a network, see reply #25  Roll Eyes
Quote
Last paragraph. Your problem seems to be your network and not at all related to QU because QU works fine here (edit after reply #39  Embarrassed) even when images are on a USB2 drive connected to the PC.
Terry
« Last Edit: May 04, 2014, 01:39:01 PM by Terry-M » Logged
Terry-M
The Honourable Metric Mann
Forum Superhero
*****
Posts: 3196



View Profile WWW
« Reply #38 on: May 04, 2014, 12:55:30 PM »

About using QU.
Quote
I have a custom 295x295mm page, print-to-file set, and I'm placing images freehand. The intent is to create pages for uploading to an online print service to make a photobook. Pages generally have around 5-6 images, just less than 6x4 (147.5x98.4 mm) or submultiples.
One "trick" you could use is to deal with 1 or 2 pages at a time and save each as a Job. In that way you will limit the number of images that have to be processed at one time and probably make it easier to organise the book. When the whole book pages are all complete you can then process each page set.
Although I think the above is the better way in your case, if you wanted to process all pages at once, save as a Session so that when recalled, you can append each session one at a time to make the complete book. Make sure the P2F is set before you recall the sessions.
I regularly make "book" or "album" pages for my own records and do use the latter method when printing more than a few pages.
Terry
Logged
admin
Administrator
Forum Superhero
*****
Posts: 3132



View Profile Email
« Reply #39 on: May 04, 2014, 01:17:21 PM »

Now sure, there are times when it's quicker to do something unnecessarily than work out whether it's necessary or not: BTDTGTTS! But I'm sorry to have to say this, but I've had the strong impression that I've almost been called a liar rather than being told that unfortunately my use case has had to suffer for the greater good.

Gosh... not what I was going for: the liar or idiot angle.  Sad  It's obvious you know what you are doing by your diagnosis/suggestions.  Since you use some of the same tools I do, I think we got hooked in just repeating what we are seeing which was different on our systems.  I worked on it last night and found something.  I was able to repeat a slowdown with 216 by putting one of my laptops on my guest network.  With things going through the network via the router (as opposed to the LAN which uses a switch), file operations were orders of magnitude slower and 216 was actually slower than 137 in that situation.  Not as slow as what you were seeing but I suppose many factors including image content could affect that.  So, having something to go on and a platform with which to test, I did a lot of things in 217 and it absolutely flies!  I added enough images to the queue to get a 10 second delay when adding/moving an image on the last page in 216 and in 217 it is instantaneous (unmeasurable via human hands).  It's in testing now.

Thanks for your feedback in this thread and for sticking with me!

Mike
Logged
steveren
Newbie
*
Posts: 24


View Profile Email
« Reply #40 on: May 04, 2014, 08:59:17 PM »

Sounds good! I look forward to trying out 217 :-)
Logged
steveren
Newbie
*
Posts: 24


View Profile Email
« Reply #41 on: May 05, 2014, 07:02:59 PM »

YAY! 217 rocks!

Well done and many thanks :-)
Logged
admin
Administrator
Forum Superhero
*****
Posts: 3132



View Profile Email
« Reply #42 on: May 05, 2014, 08:22:43 PM »

YAY! 217 rocks!

Well done and many thanks :-)

Thank you for your assistance!

Mike
Logged
Pages: 1 2 [3]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Valid XHTML 1.0! Valid CSS!