Auto merge of #46941 - ScottAbbey:freebsd-build-update, r=alexcrichton

Re-do the FreeBSD cross-builds to use Clang and libc++. Fixes #44433

Reviving #45077, from @jld:

> The main goal here is to use FreeBSD's normal libc++, instead of
> statically linking the libstdc++ packaged with GCC, because that
> libstdc++ has bugs that cause rustc to deadlock inside LLVM.
>
> But the easiest way to use libc++ is to switch the build from GCC to
> Clang, and the Clang package in the Ubuntu image already knows how to
> cross-compile (given a sysroot and preferably cross-binutils), so the
> toolchain script now uses that instead of building a custom compiler.
>
> This also de-duplicates the build-toolchain.sh script.

#45077 was close but didn't quite make it.  I rebased @jld's work off the current `master` and started with that.

I was able to determine that this Travis error (#45077 (comment)) was ultimately caused by `src/librustc_llvm/build.rs` attempting to follow a wrong value in `LLVM_STATIC_STDCPP` (#45077 (comment)).

I looked at the downstream port for FreeBSD (https://svnweb.freebsd.org/ports/head/lang/rust/) and it seems like they do not use `--enable-llvm-static-stdcpp`.

Since `libc++` is included in the FreeBSD 10+ base system, we don't need to statically link it either?

So in b989428 I have set the FreeBSD build to not actually use `LLVM_STATIC_STDCPP`.

I was able to run `./src/ci/docker/run.sh` with both `dist-i686-freebsd` and `dist-x86_64-freebsd` successfully and in about 1 minute of testing it seemed like the dist-x86_64-freebsd results worked on a FreeBSD 11 system.

It should fix #44433, which seems to be affecting many potential users.  Also FreeBSD users should be able to `./x.py build` which should help anyone who wants to upstream fixes for FreeBSD.

Questions:

Does this approach seem to be the right way to go? Do we actually really want to statically link `libc++`? (I tried that here, but it ultimately ran into a roadblock on x86_64: #45077 (comment))

Can we rewrite the comment here to be more clear about why some systems aren't going to actually use this option:
https://github.com/rust-lang/rust/blob/b989428f7dec7b52d68bed6a21e9b5b0a8086267/src/bootstrap/compile.rs#L550-L553

How does this affect users of older FreeBSD systems? It seemed like no one was complaining about using a 10.3 base version in the thread for #45077.  FreeBSD seems to only officially support 10.3, 10.4, and 11.x right now, do we have to consider older users? The `libc++` stuff came in for FreeBSD 10, older FreeBSD used `libstdc++`.

Looks like @alexcrichton was leading the discussion on the previous issue:

r? @alexcrichton

Let me know what I can do to help get this through.

src/bootstrap/compile.rs

@@ -550,6 +550,7 @@ pub fn rustc_cargo(build: &Build,
     // Building with a static libstdc++ is only supported on linux right now,
     // not for MSVC or macOS
     if build.config.llvm_static_stdcpp &&
+       !target.contains("freebsd") &&
        !target.contains("windows") &&
        !target.contains("apple") {
         cargo.env("LLVM_STATIC_STDCPP",

src/ci/docker/dist-i686-freebsd/Dockerfile

@@ -1,7 +1,7 @@
 FROM ubuntu:16.04
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-  g++ \
+  clang \
   make \
   file \
   curl \
@@ -16,16 +16,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
   libssl-dev \
   pkg-config
 
-COPY dist-i686-freebsd/build-toolchain.sh /tmp/
-RUN /tmp/build-toolchain.sh i686
+COPY scripts/freebsd-toolchain.sh /tmp/
+RUN /tmp/freebsd-toolchain.sh i686
 
 COPY scripts/sccache.sh /scripts/
 RUN sh /scripts/sccache.sh
 
 ENV \
     AR_i686_unknown_freebsd=i686-unknown-freebsd10-ar \
-    CC_i686_unknown_freebsd=i686-unknown-freebsd10-gcc \
-    CXX_i686_unknown_freebsd=i686-unknown-freebsd10-g++
+    CC_i686_unknown_freebsd=i686-unknown-freebsd10-clang \
+    CXX_i686_unknown_freebsd=i686-unknown-freebsd10-clang++
 
 ENV HOSTS=i686-unknown-freebsd
 

src/ci/docker/dist-x86_64-freebsd/Dockerfile

@@ -1,7 +1,7 @@
 FROM ubuntu:16.04
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-  g++ \
+  clang \
   make \
   file \
   curl \
@@ -16,16 +16,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
   libssl-dev \
   pkg-config
 
-COPY dist-x86_64-freebsd/build-toolchain.sh /tmp/
-RUN /tmp/build-toolchain.sh x86_64
+COPY scripts/freebsd-toolchain.sh /tmp/
+RUN /tmp/freebsd-toolchain.sh x86_64
 
 COPY scripts/sccache.sh /scripts/
 RUN sh /scripts/sccache.sh
 
 ENV \
     AR_x86_64_unknown_freebsd=x86_64-unknown-freebsd10-ar \
-    CC_x86_64_unknown_freebsd=x86_64-unknown-freebsd10-gcc \
-    CXX_x86_64_unknown_freebsd=x86_64-unknown-freebsd10-g++
+    CC_x86_64_unknown_freebsd=x86_64-unknown-freebsd10-clang \
+    CXX_x86_64_unknown_freebsd=x86_64-unknown-freebsd10-clang++
 
 ENV HOSTS=x86_64-unknown-freebsd
 

src/ci/docker/scripts/freebsd-toolchain.sh

@@ -0,0 +1,103 @@
+#!/bin/bash
+# Copyright 2016-2017 The Rust Project Developers. See the COPYRIGHT
+# file at the top-level directory of this distribution and at
+# http://rust-lang.org/COPYRIGHT.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+set -eux
+
+arch=$1
+binutils_version=2.25.1
+freebsd_version=10.3
+triple=$arch-unknown-freebsd10
+sysroot=/usr/local/$triple
+
+hide_output() {
+  set +x
+  local on_err="
+echo ERROR: An error was encountered with the build.
+cat /tmp/build.log
+exit 1
+"
+  trap "$on_err" ERR
+  bash -c "while true; do sleep 30; echo \$(date) - building ...; done" &
+  local ping_loop_pid=$!
+  $@ &> /tmp/build.log
+  trap - ERR
+  kill $ping_loop_pid
+  set -x
+}
+
+# First up, build binutils
+mkdir binutils
+cd binutils
+curl https://ftp.gnu.org/gnu/binutils/binutils-${binutils_version}.tar.bz2 | tar xjf -
+mkdir binutils-build
+cd binutils-build
+hide_output ../binutils-${binutils_version}/configure \
+  --target="$triple" --with-sysroot="$sysroot"
+hide_output make -j"$(getconf _NPROCESSORS_ONLN)"
+hide_output make install
+cd ../..
+rm -rf binutils
+
+# Next, download the FreeBSD libraries and header files
+mkdir -p "$sysroot"
+case $arch in
+  (x86_64) freebsd_arch=amd64 ;;
+  (i686) freebsd_arch=i386 ;;
+esac
+
+files_to_extract=(
+"./usr/include"
+"./usr/lib/*crt*.o"
+)
+# Try to unpack only the libraries the build needs, to save space.
+for lib in c cxxrt gcc_s m thr util; do
+  files_to_extract=("${files_to_extract[@]}" "./lib/lib${lib}.*" "./usr/lib/lib${lib}.*")
+done
+for lib in c++ c_nonshared compiler_rt execinfo gcc pthread rt ssp_nonshared; do
+  files_to_extract=("${files_to_extract[@]}" "./usr/lib/lib${lib}.*")
+done
+
+URL=https://download.freebsd.org/ftp/releases/${freebsd_arch}/${freebsd_version}-RELEASE/base.txz
+curl "$URL" | tar xJf - -C "$sysroot" --wildcards "${files_to_extract[@]}"
+
+# Fix up absolute symlinks from the system image.  This can be removed
+# for FreeBSD 11.  (If there's an easy way to make them relative
+# symlinks instead, feel free to change this.)
+set +x
+find "$sysroot" -type l | while read symlink_path; do
+  symlink_target=$(readlink "$symlink_path")
+  case $symlink_target in
+    (/*)
+      echo "Fixing symlink ${symlink_path} -> ${sysroot}${symlink_target}" >&2
+      ln -nfs "${sysroot}${symlink_target}" "${symlink_path}" ;;
+  esac
+done
+set -x
+
+# Clang can do cross-builds out of the box, if we give it the right
+# flags.  (The local binutils seem to work, but they set the ELF
+# header "OS/ABI" (EI_OSABI) field to SysV rather than FreeBSD, so
+# there might be other problems.)
+#
+# The --target option is last because the cross-build of LLVM uses
+# --target without an OS version ("-freebsd" vs. "-freebsd10").  This
+# makes Clang default to libstdc++ (which no longer exists), and also
+# controls other features, like GNU-style symbol table hashing and
+# anything predicated on the version number in the __FreeBSD__
+# preprocessor macro.
+for tool in clang clang++; do
+  tool_path=/usr/local/bin/${triple}-${tool}
+  cat > "$tool_path" <<EOF
+#!/bin/sh
+exec $tool --sysroot=$sysroot --prefix=${sysroot}/bin "\$@" --target=$triple
+EOF
+  chmod +x "$tool_path"
+done