Ticket #951 (new enhancement)

Opened 7 years ago

Last modified 6 years ago

ability to reinstall packages depending upon given spec

Reported by: filko_ Owned by:
Priority: Sometime Milestone: cave Long Term Extras
Component: clients/cave Version: 0.50.3
Keywords: Cc: Ivan.Miljenovic@…, markus@…
Blocked By: Blocking:
Distribution: Gentoo


Possibly --needing or --requiring

Change History

comment:1 Changed 7 years ago by ciaranm

  • Milestone changed from cave Useful Functionality to cave Long Term Extras

comment:2 Changed 7 years ago by ivanm

  • Cc Ivan.Miljenovic@… added

Requirements for Haskell libraries:

Due to Haskell libraries being managed by GHC, each install of a library has a unique ABI based in part on its dependencies.

Consider the following scenario:

  • Library b depends upon library a, and library c depends upon b.
  • Library a is upgraded/downgraded (to a version that still satisfies b's dependencies).
  • Library b is now in an inconsistent state (as reported by ghc-pkg check) and needs to be rebuilt. However, to guarantee no failures library c should also be rebuilt (which ghc-pkg check will report). Note that if b is rebuilt on its own then ghc-pkg check will not complain that c needs to be rebuilt.

I am not sure how much of this situation regarding c is required and how much is just a case of institutional wisdom regarding this (since that's what's been done all along). However, the general consensus currently is that for all Haskell libraries, if a (dependency of a)+ dependency is upgraded/downgraded, then it should be as well. I _think_ this is because in the case above the API of library b may expose parts of as API, and as such it's API/ABI has changed as well.

As of GHC 6.12.1, dynamic linking is now supported for executables; however at least in Gentoo we have not yet enabled support for dynamically linked executables (and I'm not sure of any Linux distribution that has), so I am unsure if such executables would need to be rebuilt as well.

comment:3 Changed 7 years ago by dleverton

18:08 < dleverton_> ciaranm: so for that Haskell stuff ivanm was talking about, I was thinking of:
18:09 < dleverton_> ciaranm: have an optional phase called at the end of the build that can return some kind of "ABI identifier" - for haskell 
                    packages it can just return the hash GHC generates - and store it in the exndbam
18:11 < dleverton_> ciaranm: when rebuilding/upgrading/downgrading such a package in the same slot, bring everything that depends on it (possibly 
                    triggered by some new dep syntax similar to :=) into the resolution as "might need to be rebuilt"
18:12 < dleverton_> ciaranm: after the new version of the library is built, if the ABI identifier changed, actually go ahead and rebuild all those 
                    packages, otherwise skip them (similar to your idea for SCM packages where they're flagged as "maybe" at resolution time, then 
                    rebuilt or skipped depending on whether there are any changes in the repository)
18:12 < dleverton_> ciaranm: does that sound somewhat sensible?
18:15 < dleverton_> ciaranm: (and if a being rebuilt causes b to be a "maybe", that should recursively pull in c as a maybe too)
18:44 < ciaranm> dleverton_: that sounds extremely complicated
18:45 < dleverton_> Really?
18:46 < ciaranm> the whole 'maybe' thing, yeah
18:46 < dleverton_> It's not massively different from the SCM thing - check whether some external thing has changed since the package was last built, 
                    and rebuild if so
18:46 < ciaranm> the scm thing is also difficult!
18:46 < dleverton_> Do you still want to do that eventually?
18:46 < ciaranm> in theory
18:46 < ciaranm> could mmmmmaybe do it by making taken() something other than a bool
18:46 < ciaranm> taken = yes | probably | no or something
18:47 < ciaranm> not sure how it'd interact with continue-on-ahf etc
18:47 < dleverton_> You mean what do if the package gets skipped after all?
18:47 < ciaranm> might need a SkippedBecauseUnnecessary state or something
18:47 < dleverton_> Yeah
18:48 < ciaranm> the other thing for scm... it'd be a lot easier if the extra fetch parts were backgroundable
18:51 < dleverton_> Anyway, other than implementation complexity, does it sound reasonable?
18:51 < ciaranm> anyway. making the api usable first!
18:51 < ciaranm> it doesn't sound obviously not doable
18:51 < dleverton_> Want it on the ticket you pointed ivanm at? (which I thought was for a user feature, not a package format one)
18:51 < ciaranm> i worry that it'll end up with cave resolve prettymuchanysmallpackage ending up 'maybe' rebuilding the entire tree
18:52 < ciaranm> yes please
18:52 < ciaranm> i'm not sure whether it *should* be a user feature or a package format feature
18:52 < ciaranm> or both

comment:4 Changed 7 years ago by ivanm

According to dcoutts (main developer of the Cabal build library for Haskell and an ex-Gentoo dev), the problem here is assuming that the ABI corresponds to the version number, when in fact it doesn't. If we assume GHC >= 6.12, then a better definition of the ABI would be the hash that ghc-pkg stores (specifically, "ghc-pkg field <library> id").

So, is there any way we can 1) grab the hash from ghc-pkg of all dependencies and store them at build-time, and then 2) use that hash as an ABI dependency indicator?

dcoutts says that this will also be a problem if dynamic linking is used for executables with GHC >= 6.12 rather than the current default of static linking.

comment:5 Changed 7 years ago by ivanm

Note that the hash _may_ change any time you rebuild a library; if it doesn't then you don't need to rebuild anything that depends upon it.

comment:6 Changed 7 years ago by ciaranm

I think this ticket is now covering two different and largely unrelated things.

On the one hand, we want a user control for it, sort of like fix-linkage --library, so you could do cave resolve --dependent-upon blah or something.

On the other hand, we want a package control thing, to say that "when I'm changed, blah will break".

I don't think these are the same feature.

comment:7 Changed 6 years ago by mrothe

  • Cc markus@… added

comment:8 Changed 6 years ago by pipping

sci-apps/maxima::pipping in exherbo has a similar problem.

maxima has sbcl load maxima and then save state; the resulting file is called a dump.

sbcl will refuse to load a dump made with another instance of sbcl; even if one does not touch the exheres, a new installation of sbcl will refuse to read dumps from another installation.

This means that maxima has to be rebuilt everytime sbcl is rebuilt. The spec of sbcl is irrelevant.

Last edited 6 years ago by pipping (previous) (diff)

comment:9 Changed 6 years ago by dleverton

How does SBCL identify that the dump is or isn't compatible? Does it embed a hash of itself / timestamp when it was built / etc into the dump?

comment:10 Changed 6 years ago by pipping

Ignore that I claimed those files were called dumps. They're called cores.

Here's a quote from the documentation of (save-lisp-and-die) from sbcl/src/code/save.lisp:

There is absolutely no binary compatibility of core images between different runtime support programs. Even runtimes built from the same sources at different times are treated as incompatible for this purpose.

As for how that is determined... Here's the relevant snippet from sbcl/src/runtime/codeparse.c:

                unsigned int i;

                FSHOW((stderr, "build_id[]=\"%s\"\n", build_id));
                FSHOW((stderr, "remaining_len = %d\n", remaining_len));
                if (remaining_len != strlen((const char *)build_id))
                    goto losing_build_id;
                for (i = 0; i < remaining_len; ++i) {
                    FSHOW((stderr, "ptr[%d] = char = %d, expected=%d\n",
                           i, ptr[i], build_id[i]));
                    if (ptr[i] != build_id[i])
                        goto losing_build_id;
                /* .core files are not binary-compatible between
                 * builds because we can't easily detect whether the
                 * sources were patched between the time the
                 * dumping-the-.core runtime was built and the time
                 * that the loading-the-.core runtime was built.
                 * (We could easily detect whether version.lisp-expr
                 * was changed, but people experimenting with patches
                 * don't necessarily update version.lisp-expr.) */

                lose("can't load .core for different runtime, sorry\n");

which leaves the question how that build id is determined. Here--I think---is how, from make-config.sh:

  echo '"'`hostname`-`id -un`-`date +%Y-%m-%d-%H-%M-%S`'"' > output/build-id.tmp

comment:11 Changed 6 years ago by dleverton

OK, then that build_id would basically be treated as the ABI "version".

(For the record, since it's not already mentioned on this ticket,  http://lists.exherbo.org/pipermail/exherbo-dev/2011-March/000891.html is the same idea as the IRC log above, but possibly more comprehensive.)

Note: See TracTickets for help on using tickets.