Ticket #76 (assigned defect)

Opened 11 years ago

Last modified 3 years ago

paludis --list-packages / cave print-packages actions are too slow

Reported by: pioto Owned by:
Priority: Sometime Milestone:
Component: clients/paludis Version: 0.16.2
Keywords: Cc:
Blocked By: Blocking:
Distribution: Gentoo

Description (last modified by pioto) (diff)

It's too slow to use for bashcomp and such, so I gotta use some ugly hacks instead to get reasonable speed.

I need to figure out how to properly profile it.

Change History

comment:1 Changed 11 years ago by ciaranm

Hrm. Maybe we should look into reference counting Validated. Not sure that it would help though...

comment:2 Changed 10 years ago by ferdy

This is a simple profile I just took:  http://dev.gentoo.org/~ferdy/tmp/list_packages_profile

Also, my feeling while reading and studying the paludis' codebase was that we did too many stat calls:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 39.41    0.018231           0     51100      9393 lstat64
 19.94    0.009225           0     37159           write
 14.64    0.006772           0     25329           getdents
 13.77    0.006373           0     27656         8 open
  7.68    0.003552           0     30566           read
  2.29    0.001061           0     27648           close
  1.26    0.000581          39        15           mprotect
  0.54    0.000252           0     12670           fstat64
  0.35    0.000164           0     12652           fcntl64
  0.07    0.000032          16         2           munmap
  0.05    0.000022           0       228           brk
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         7           time
  0.00    0.000000           0         1           getpid
  0.00    0.000000           0         1           access
  0.00    0.000000           0         2           ioctl
  0.00    0.000000           0        19           readlink
  0.00    0.000000           0        38           mmap2
  0.00    0.000000           0         8         5 stat64
  0.00    0.000000           0         1           set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00    0.046265                225103      9406 total

That'd be for listing ~13k packages.

Unfortunately, those numbers are with my work in progress of doing inodesort in hot places. I have some more DirIterator?+FSEntry optimizations in my head that would save us _some_ stat calls since some info is available when we run readdir.

  • ferdy

comment:3 Changed 9 years ago by pioto

  • Description modified (diff)

Just to keep this up-to-date-ish... right now I'm getting about 15 seconds to do a --list-packages with a hot cache, 1 minutes with a cold cache...

If I limit it just to a few repositories, like all the installed ones, then it only takes a few seconds. So, it might make sense to use --list-packages for completion when we're --uninstalling... but otherwise, I think it is still too slow.

comment:4 Changed 9 years ago by pioto

  • Distribution set to Gentoo
  • Summary changed from --list-packages action is too slow to paludis --list-packages / cave print-packages actions are too slow

As a further note to myself... this tutorial on gprof looks reasonable...  http://linuxgazette.net/100/vinayak.html

Currently, on a different system, cave print-packages w/ a hot cache is on the order of 5 seconds, and about 20 seconds with a cold cache.

I still think it might be acceptable to use some hacks with the names cache, if it exists, to solve this problem. However, I'd like to run a profile on the current state of things first, to get a more solid idea of what's up.

comment:5 Changed 9 years ago by ciaranm

Using the names cache is almost certainly wrong. If I/O is the problem, it's only a problem once. If it's not I/O, the names cache won't make much difference, and the solution is to be cleverer in how we manage whatever the slowest part is internally.

comment:6 Changed 9 years ago by ciaranm

Try removing the "has ebuilds" check in TraditionalLayout::package_names and replacing it with a hard-coded blacklist of "directory is not called CVS". Does that help?

comment:7 Changed 9 years ago by pioto

  • Status changed from new to accepted

Forward progress made with eb990eb. Dunno if I want to mark this as FIXED yet.

comment:8 Changed 3 years ago by dleverton

  • Status changed from accepted to assigned
  • Owner pioto deleted
Note: See TracTickets for help on using tickets.