[vlc-devel] [PATCH v2 13/13] extras: extract the contrib sources before building them
Alexandre Janniaux
ajanni at videolabs.io
Fri Jul 10 15:02:24 CEST 2020
Hi,
On Fri, Jul 10, 2020 at 01:08:40PM +0200, Steve Lhomme wrote:
> On 2020-07-10 12:26, Alexandre Janniaux wrote:
> > Hi,
> >
> > On Fri, Jul 10, 2020 at 11:30:38AM +0200, Steve Lhomme wrote:
> > > On 2020-07-10 11:06, Alexandre Janniaux wrote:
> > > > Hi,
> > > >
> > > > The point of Remi, that I agree on, is that you're not
> > > > supposed to follow contrib build lines, or you're supposed
> > >
> > > First time I hear that logs should not be read. If that's the case we can
> > > redirect them to /dev/null
> >
> > Read doesn't mean follow. You can easily spot which contrib failed
> > and just run the build of the very contrib that failed.
>
> No I can't *just* run the build. I have nothing to compile for macOS or iOS.
> I need to be able to read the logs when there's something wrong.
I compile both on my Linux machine, although macOS is still
a bit unconvenient to compile. But if you don't want to fiddle
about the setup, then in this very case it is mainly an issue
within the CI, not an issue with the contribs.
Even with changing the CI build script, if the contrib cannot
be tested for all platform at once, it should probably be
enabled only for the platform you can test, and then enabled
platform by platform by other developers who can test on their
side and be able to actually test the end result.
CI resources are quite scarce and expensive so using them to
develop is slow, inefficient resource-wise and has an impact
on other developer's build which are only validation.
Regards,
--
Alexandre Janniaux
Videolabs
>
> > >
> > > > to build with -j1 if that what you really want.
> > > >
> > > > I rarely needed to follow the build too, I usually just go
> > > > by backward search "ERROR 1" or "ERROR 2" and typically
> > > > directly find the relevant issue.
> > >
> > > I do that too. But when the problem is a patch not applying properly, good
> > > luck finding it.
> >
> > Likewise, the CI should refuse any patch breaking cntribs and
> > it's expected that all patch are applying correctly, so the
> > real issue is local development, in which case you should not
> > build every contribs but just the one you're trying to patch,
> > and probably use make -j1 in this case.
>
> That's fine for code that is merged. But I do a lot of back and forth
> changes in contribs before reaching something that works everywhere (gnutls
> was particularly painful). The patches you see in the ML and the git are
> just the tip of the iceberg. There's a lot of work done before a patch can
> work on all platforms. And for that we need proper debugging tools.
>
> > Regards,
> > --
> > Alexandre Janniaux
> > Videolabs
> >
> > >
> > > > Regards,
> > > > --
> > > > Alexandre Janniaux
> > > > Videolabs
> > > >
> > > > On Fri, Jul 10, 2020 at 10:24:43AM +0200, Steve Lhomme wrote:
> > > > > On 2020-07-09 15:24, Alexandre Janniaux wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I agree with Remi, there's no point in splitting both for
> > > > > > most users, and it doesn't even solve the issue with multiple
> > > > > > jobs because you could have an issue with one package, but the
> > > > > > other jobs continuing to output text afterwards.
> > > > >
> > > > > It's not meant to solve all problems but it's a step in the right direction.
> > > > > Extracting sources typically output a ton more lines per minute than the
> > > > > building phase. So if you happen to have extraction during the build phase
> > > > > you can't follow at all the build lines.
> > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Alexandre Janniaux
> > > > > > Videolabs
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 09, 2020 at 02:43:44PM +0200, Steve Lhomme wrote:
> > > > > > > On 2020-07-07 19:13, Rémi Denis-Courmont wrote:
> > > > > > > > Logs are confusing if you use multiple jobs. This has nothing to do with
> > > > > > > > tar or fetch.
> > > > > > >
> > > > > > > Something is bad so we shouldn't do anything to fix it even halfway. That's
> > > > > > > simply wrong.
> > > > > > >
> > > > > > > > -1.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Le 7 juillet 2020 08:29:34 GMT+02:00, Steve Lhomme <robux4 at ycbcr.xyz> a
> > > > > > > > écrit :
> > > > > > > >
> > > > > > > > This will make the logs easier to follow, especially when an error occurs.
> > > > > > > >
> > > > > > > > The fetch part might be removed but the downloads will end up being mixed with
> > > > > > > > the package untaring which is also unreadable.
> > > > > > > > ------------------------------------------------------------------------
> > > > > > > > extras/ci/gitlab-ci.yml | 1 +
> > > > > > > > extras/package/apple/build.sh | 3 +++
> > > > > > > > extras/package/macosx/build.sh | 1 +
> > > > > > > > extras/package/raspberry/build.sh | 1 +
> > > > > > > > extras/package/snap/snapcraft.yaml | 1 +
> > > > > > > > extras/package/win32/build.sh | 1 +
> > > > > > > > 6 files changed, 8 insertions(+)
> > > > > > > >
> > > > > > > > diff --git a/extras/ci/gitlab-ci.yml b/extras/ci/gitlab-ci.yml
> > > > > > > > index 61116fc26b9..3ac785d7283 100644
> > > > > > > > --- a/extras/ci/gitlab-ci.yml
> > > > > > > > +++ b/extras/ci/gitlab-ci.yml
> > > > > > > > @@ -198,6 +198,7 @@ debian:
> > > > > > > > make prebuilt PREBUILT_URL="${VLC_PREBUILT_CONTRIBS_URL}"
> > > > > > > > else
> > > > > > > > make -j$NCPU --output-sync=recurse fetch
> > > > > > > > + make -j$NCPU --output-sync=recurse extract
> > > > > > > > make -j$NCPU --output-sync=recurse
> > > > > > > > make package
> > > > > > > > fi
> > > > > > > > diff --git a/extras/package/apple/build.sh b/extras/package/apple/build.sh
> > > > > > > > index 1b112e84651..97c3e8455d0 100755
> > > > > > > > --- a/extras/package/apple/build.sh
> > > > > > > > +++ b/extras/package/apple/build.sh
> > > > > > > > @@ -611,6 +611,9 @@ else
> > > > > > > > # Download source packages
> > > > > > > > $MAKE fetch -j$VLC_USE_NUMBER_OF_CORES
> > > > > > > > + # Extract source packages
> > > > > > > > + $MAKE extract -j$VLC_USE_NUMBER_OF_CORES
> > > > > > > > +
> > > > > > > > # Build contribs
> > > > > > > > $MAKE -j$VLC_USE_NUMBER_OF_CORES || abort_err "Building contribs failed"
> > > > > > > > diff --git a/extras/package/macosx/build.sh b/extras/package/macosx/build.sh
> > > > > > > > index 69395ef5d0d..90e06b31942 100755
> > > > > > > > --- a/extras/package/macosx/build.sh
> > > > > > > > +++ b/extras/package/macosx/build.sh
> > > > > > > > @@ -166,6 +166,7 @@ mkdir -p contrib-$TRIPLET && cd contrib-$TRIPLET
> > > > > > > > if [ "$CONTRIBFROMSOURCE" = "yes" ]; then
> > > > > > > > make list
> > > > > > > > make fetch
> > > > > > > > + make extract
> > > > > > > > make -j$JOBS .gettext
> > > > > > > > make -j$JOBS -k || make -j1
> > > > > > > > diff --git a/extras/package/raspberry/build.sh b/extras/package/raspberry/build.sh
> > > > > > > > index a3c4dff2f2d..bb86c4e622a 100755
> > > > > > > > --- a/extras/package/raspberry/build.sh
> > > > > > > > +++ b/extras/package/raspberry/build.sh
> > > > > > > > @@ -127,6 +127,7 @@ export LDFLAGS="$LDFLAGS -L/usr/$TRIPLET/lib -L/opt/vc/lib"
> > > > > > > > if [ "$PREBUILT" != "yes" ]; then
> > > > > > > > make list
> > > > > > > > make -j$JOBS --output-sync=recurse fetch
> > > > > > > > + make -j$JOBS --output-sync=recurse extract
> > > > > > > > make -j$JOBS --output-sync=recurse -k || make -j1
> > > > > > > > if [ "$PACKAGE" = "yes" ]; then
> > > > > > > > make package
> > > > > > > > diff --git a/extras/package/snap/snapcraft.yaml b/extras/package/snap/snapcraft.yaml
> > > > > > > > index 7ee9954cf33..dce31d1d2a7 100644
> > > > > > > > --- a/extras/package/snap/snapcraft.yaml
> > > > > > > > +++ b/extras/package/snap/snapcraft.yaml
> > > > > > > > @@ -84,6 +84,7 @@ parts:
> > > > > > > > else
> > > > > > > > make list
> > > > > > > > make -j $(getconf _NPROCESSORS_ONLN) fetch
> > > > > > > > + make -j $(getconf _NPROCESSORS_ONLN) extract
> > > > > > > > make -j $(getconf _NPROCESSORS_ONLN) -k install || make -j1
> > > > > > > > make package
> > > > > > > > fi
> > > > > > > > diff --git a/extras/package/win32/build.sh b/extras/package/win32/build.sh
> > > > > > > > index 568159907ba..c30cfc65222 100755
> > > > > > > > --- a/extras/package/win32/build.sh
> > > > > > > > +++ b/extras/package/win32/build.sh
> > > > > > > > @@ -300,6 +300,7 @@ ${VLC_ROOT_PATH}/contrib/bootstrap --host=$TRIPLET --prefix=../$CONTRIB_PREFIX $
> > > > > > > > if [ "$PREBUILT" != "yes" ]; then
> > > > > > > > make list
> > > > > > > > make -j$JOBS fetch
> > > > > > > > + make -j$JOBS extract
> > > > > > > > make -j$JOBS -k || make -j1
> > > > > > > > if [ "$PACKAGE" = "yes" ]; then
> > > > > > > > make package
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser
> > > > > > > > ma brièveté.
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > vlc-devel mailing list
> > > > > > > > To unsubscribe or modify your subscription options:
> > > > > > > > https://mailman.videolan.org/listinfo/vlc-devel
> > > > > > > >
> > > > > > > _______________________________________________
> > > > > > > vlc-devel mailing list
> > > > > > > To unsubscribe or modify your subscription options:
> > > > > > > https://mailman.videolan.org/listinfo/vlc-devel
> > > > > > _______________________________________________
> > > > > > vlc-devel mailing list
> > > > > > To unsubscribe or modify your subscription options:
> > > > > > https://mailman.videolan.org/listinfo/vlc-devel
> > > > > >
> > > > > _______________________________________________
> > > > > vlc-devel mailing list
> > > > > To unsubscribe or modify your subscription options:
> > > > > https://mailman.videolan.org/listinfo/vlc-devel
> > > > _______________________________________________
> > > > vlc-devel mailing list
> > > > To unsubscribe or modify your subscription options:
> > > > https://mailman.videolan.org/listinfo/vlc-devel
> > > >
> > > _______________________________________________
> > > vlc-devel mailing list
> > > To unsubscribe or modify your subscription options:
> > > https://mailman.videolan.org/listinfo/vlc-devel
> > _______________________________________________
> > vlc-devel mailing list
> > To unsubscribe or modify your subscription options:
> > https://mailman.videolan.org/listinfo/vlc-devel
> >
> _______________________________________________
> vlc-devel mailing list
> To unsubscribe or modify your subscription options:
> https://mailman.videolan.org/listinfo/vlc-devel
More information about the vlc-devel
mailing list