From a54d1edf960ad04bb6d31424181c58621e26b8ca Mon Sep 17 00:00:00 2001 From: Austin Anderson <angerson@google.com> Date: Tue, 15 Dec 2020 13:20:07 -0800 Subject: [PATCH] Rearrange files and projects to match #16 (#17) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🎉 --- .github/ISSUE_TEMPLATE.md | 16 -------- .github/PULL_REQUEST_TEMPLATE.md | 6 --- CODEOWNERS | 9 +++++ CONTRIBUTING.md | 16 +++++++- README.md | 39 ++++++++++++++++--- directory_template/README.md | 21 ++++++++++ images/Dockerfile.linux.cpu | 26 ------------- images/README.md | 37 ------------------ .../Dockerfile.manylinux2014.cpu | 0 manylinux2014_docker_images/README.md | 23 +++++++++++ .../install/bazelrc | 0 .../install/install_bazel.sh | 0 .../install/install_buildifier.sh | 0 .../install/install_deb_packages.sh | 0 .../install/install_golang.sh | 0 .../install/install_pip_packages.sh | 0 .../install/install_proto3.sh | 0 .../install/install_rpm_packages.sh | 0 ppc64le_builds/README.md | 19 +++++++++ .../images}/cpu/Dockerfile.manylinux_2014 | 0 .../gpu/Dockerfile.manylinux_2014.cuda10_1 | 0 .../images}/gpu/cuda.repo | 0 .../scripts}/cpu_build.sh | 0 .../scripts}/gpu_build.sh | 0 tekton/README.md | 9 +++++ {webhook => tekton/webhook}/Dockerfile | 0 {webhook => tekton/webhook}/app.py | 0 {webhook => tekton/webhook}/config.py | 0 {webhook => tekton/webhook}/requirements.txt | 0 29 files changed, 128 insertions(+), 93 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE.md delete mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 100644 CODEOWNERS create mode 100644 directory_template/README.md delete mode 100644 images/Dockerfile.linux.cpu delete mode 100644 images/README.md rename {images => manylinux2014_docker_images}/Dockerfile.manylinux2014.cpu (100%) create mode 100644 manylinux2014_docker_images/README.md rename {images => manylinux2014_docker_images}/install/bazelrc (100%) rename {images => manylinux2014_docker_images}/install/install_bazel.sh (100%) rename {images => manylinux2014_docker_images}/install/install_buildifier.sh (100%) rename {images => manylinux2014_docker_images}/install/install_deb_packages.sh (100%) rename {images => manylinux2014_docker_images}/install/install_golang.sh (100%) rename {images => manylinux2014_docker_images}/install/install_pip_packages.sh (100%) rename {images => manylinux2014_docker_images}/install/install_proto3.sh (100%) rename {images => manylinux2014_docker_images}/install/install_rpm_packages.sh (100%) create mode 100644 ppc64le_builds/README.md rename {images/ppc64le => ppc64le_builds/images}/cpu/Dockerfile.manylinux_2014 (100%) rename {images/ppc64le => ppc64le_builds/images}/gpu/Dockerfile.manylinux_2014.cuda10_1 (100%) rename {images/ppc64le => ppc64le_builds/images}/gpu/cuda.repo (100%) rename {ci_environments/ppc64le => ppc64le_builds/scripts}/cpu_build.sh (100%) rename {ci_environments/ppc64le => ppc64le_builds/scripts}/gpu_build.sh (100%) create mode 100644 tekton/README.md rename {webhook => tekton/webhook}/Dockerfile (100%) rename {webhook => tekton/webhook}/app.py (100%) rename {webhook => tekton/webhook}/config.py (100%) rename {webhook => tekton/webhook}/requirements.txt (100%) diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md deleted file mode 100644 index 3c52212..0000000 --- a/.github/ISSUE_TEMPLATE.md +++ /dev/null @@ -1,16 +0,0 @@ -## Expected Behavior - - -## Actual Behavior - - -## Steps to Reproduce the Problem - -1. -1. -1. - -## Specifications - -- Version: -- Platform: \ No newline at end of file diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md deleted file mode 100644 index 0787bd9..0000000 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ /dev/null @@ -1,6 +0,0 @@ -Fixes #<issue_number_goes_here> - -> It's a good idea to open an issue first for discussion. - -- [ ] Tests pass -- [ ] Appropriate changes to README are included in PR \ No newline at end of file diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 0000000..df1cd6e --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1,9 @@ +# SIG Leads are default for PRs unless a specific owner is listed below +* @angerson @perfinion + +# These should match the Maintainer: line in your README.md +# Please sort them in alphabetical order. +/directory_template @angerson @perfinion +/manylinux_2014_docker_images @sub-mod +/ppc64le @wdirons +/tekton @perfinion diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5fab2c1..d15f863 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,19 @@ # How to Contribute -We'd love to accept your patches and contributions to this project. There are -just a few small guidelines you need to follow. +To add your own project to our showcase, you can: + +* Add a link and a short description to [README.md](README.md) in the same + format as the other links, and then file a PR, or, +* Copy the [Directory Template](directory_template) into a new folder, fill in + the template, and add it to the showcase. Make sure to update + [CODEOWNERS](CODEOWNERS) as well. + +This repo is a good place for small additions, but if the project is a big one, +it may be better to maintain it in your own repository for better encapsulation +of responsibility (and so you can make your own rules for further +contributions). + +There are just a few small standard guidelines you need to follow as well: ## Contributor License Agreement diff --git a/README.md b/README.md index f011d00..f5e4fa8 100644 --- a/README.md +++ b/README.md @@ -7,22 +7,49 @@ [](https://gitter.im/tensorflow/sig-build) [](https://groups.google.com/a/tensorflow.org/g/build) -**TensorFlow SIG Build** is a community group dedicated to improving all aspects -of the TensorFlow build process. +**TensorFlow SIG Build** is a community group dedicated to the TensorFlow build +process. This repository is a showcase of resources, guides, tools, and builds +contributed by the community, for the community. -## Contributing +## Group -SIG Build is a community led open source project. As such, the project +### Contributing + +SIG Build is a community-led open source project. As such, the project depends on public contributions, bug-fixes, and documentation. Please see [contribution guidelines](CONTRIBUTING.md) for a guide on how to contribute. This project adheres to [TensorFlow's code of conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. -## Community +### Community * [Public Mailing List](https://groups.google.com/a/tensorflow.org/forum/#!forum/build) * [SIG Monthly Meeting Notes](https://docs.google.com/document/d/10_3IQ5aF-88ADJNLF0WOpb09bZ15x-sBnRSnDHNCNr8/edit) * Join our mailing list and receive calendar invites to the meeting. -## License +### License [Apache License 2.0](LICENSE) + +## Project Showcase + +Want to add your own project to this list? It's easy: check out +[CONTRIBUTING.md](CONTRIBUTING.md). + +### Docker + +* [**Manylinux 2014 Docker Images**](manylinux_2014_docker_images): + `manylinux2014` build environment for TensorFlow packages. +* [**Distroless Dockerfiles**](https://github.com/uvarc/rivanna-docker): + Distroless ([info](https://github.com/GoogleContainerTools)) TensorFlow + images, which are smaller than TensorFlow's official images. + +### Platforms + +* [**ppc64le Builds**](ppc64le_builds): Dockerfiles and wheel build scripts for + building TF on ppc64le. + +### WIP / Other + +* [**Tekton CI**](tekton): perfinion's experimental directory for using Tekton + CI with TensorFlow +* [**Directory Template**](directory_template): Example short description. diff --git a/directory_template/README.md b/directory_template/README.md new file mode 100644 index 0000000..91c9e40 --- /dev/null +++ b/directory_template/README.md @@ -0,0 +1,21 @@ +# Directory Template + +Example short description. + +Maintainer: @angerson (TensorFlow, SIG Build) + +* * * + +This is is a sample of a new directory for showcasing a local project. Anything +goes, just please follow these rules to keep things straightforward: + +- Name the directory the same as the title `in_snake_case` (all lowercase, with + underscores, no hyphens). +- Your directory should have a README.md with a header: + - Title identical to the one added on the root README.md + - Short description identical to the one added on the root README.md + - Maintainer: `<your email(s)> (<your organization(s)>)`. If you have no + organization or not acting on behalf of one, write "Independent". +- You can add this directory to the root README.md list under any Showcase + section. If it doesn't fit under any section, just put it at the top under + "Showcase." diff --git a/images/Dockerfile.linux.cpu b/images/Dockerfile.linux.cpu deleted file mode 100644 index 52f5186..0000000 --- a/images/Dockerfile.linux.cpu +++ /dev/null @@ -1,26 +0,0 @@ -FROM ubuntu:16.04 - -LABEL maintainer="Jason Zaman <jason@perfinion.com>" - -# Copy and run each install script one at a time for better layer caching. - -COPY install/install_deb_packages.sh /install/ -RUN /install/install_deb_packages.sh - -COPY install/install_golang.sh /install/ -RUN /install/install_golang.sh - -COPY install/install_proto3.sh /install/ -RUN /install/install_proto3.sh - -COPY install/install_buildifier.sh /install/ -RUN /install/install_buildifier.sh - -COPY install/install_bazel.sh /install/ -RUN /install/install_bazel.sh - -COPY install/install_pip_packages.sh /install/ -RUN /install/install_pip_packages.sh - -# Set up the master bazelrc configuration file. -COPY install/bazelrc /etc/bazel.bazelrc diff --git a/images/README.md b/images/README.md deleted file mode 100644 index 8aa43bb..0000000 --- a/images/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# Dockerfiles for TensorFlow builds - -## manylinux2014 builds -Run the below commands to create manylinux2014 build environment. -``` -docker build -t build/manylinux2014 -f Dockerfile.manylinux2014.cpu . -docker run -it build/manylinux2014:latest /bin/bash -``` - -Run the below commands to build TF. -``` -source scl_source enable devtoolset-7 rh-python36 -git clone --branch=r2.1 --depth=1 https://github.com/tensorflow/tensorflow.git -cd tensorflow/ -./configure -bazel build -c opt --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --verbose_failures //tensorflow/tools/pip_package:build_pip_package -``` - -## Distroless Images -The [Distroless](https://github.com/GoogleContainerTools/distroless) base images developed by Google are meant to contain an application and its runtime dependencies only. Please visit their GitHub for more information. - -Distroless Dockerfiles for TensorFlow CPU (`Dockerfile.cpu`) and GPU (`Dockerfile.distroless`) are hosted in the [rivanna-docker repository](https://github.com/uvarc/rivanna-docker). Using the [MNIST benchmark](https://www.tensorflow.org/tutorials/quickstart/beginner), the containers were tested on UVA's HPC Rivanna cluster to have the same performance as the official ones. - -Image size comparison: -| [Official](https://hub.docker.com/r/tensorflow/tensorflow/tags) | Size | [Distroless](https://hub.docker.com/r/uvarc/tensorflow/tags) | Size | Reduction | -|---|---|---|---|---| -| `2.3.0` | 582 MB | `2.3.0-cpu` | 227 MB | 61% | -| `2.3.0-gpu` | 1.44 GB | `2.3.0-distroless` | 1.18 GB | 18% | -| `nightly-gpu` | 2.27 GB | `2.4.0-distroless` | 1.47 GB | 35% | - -Pull command: -``` -docker pull uvarc/tensorflow:<tag> -``` - -### Contact -Research Computing at the University of Virginia (external) maintains Distroless Dockerfiles for TensorFlow in the [rivanna-docker repository](https://github.com/uvarc/rivanna-docker). Please reach out to hpc-support@virginia.edu for questions and feedback. diff --git a/images/Dockerfile.manylinux2014.cpu b/manylinux2014_docker_images/Dockerfile.manylinux2014.cpu similarity index 100% rename from images/Dockerfile.manylinux2014.cpu rename to manylinux2014_docker_images/Dockerfile.manylinux2014.cpu diff --git a/manylinux2014_docker_images/README.md b/manylinux2014_docker_images/README.md new file mode 100644 index 0000000..5981858 --- /dev/null +++ b/manylinux2014_docker_images/README.md @@ -0,0 +1,23 @@ +# Manylinux2014 Docker Images + +`manylinux2014` build environment for TensorFlow packages. + +Maintainer: @sub-mod (Red Hat) + +* * * + +Run the below commands to create a manylinux2014 build environment. +``` +docker build -t build/manylinux2014 -f Dockerfile.manylinux2014.cpu . +docker run -it build/manylinux2014:latest /bin/bash +``` + +Run the below commands to build TF. +``` +source scl_source enable devtoolset-7 rh-python36 +git clone --branch=r2.1 --depth=1 https://github.com/tensorflow/tensorflow.git +cd tensorflow/ +./configure +bazel build -c opt --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --verbose_failures //tensorflow/tools/pip_package:build_pip_package +``` + diff --git a/images/install/bazelrc b/manylinux2014_docker_images/install/bazelrc similarity index 100% rename from images/install/bazelrc rename to manylinux2014_docker_images/install/bazelrc diff --git a/images/install/install_bazel.sh b/manylinux2014_docker_images/install/install_bazel.sh similarity index 100% rename from images/install/install_bazel.sh rename to manylinux2014_docker_images/install/install_bazel.sh diff --git a/images/install/install_buildifier.sh b/manylinux2014_docker_images/install/install_buildifier.sh similarity index 100% rename from images/install/install_buildifier.sh rename to manylinux2014_docker_images/install/install_buildifier.sh diff --git a/images/install/install_deb_packages.sh b/manylinux2014_docker_images/install/install_deb_packages.sh similarity index 100% rename from images/install/install_deb_packages.sh rename to manylinux2014_docker_images/install/install_deb_packages.sh diff --git a/images/install/install_golang.sh b/manylinux2014_docker_images/install/install_golang.sh similarity index 100% rename from images/install/install_golang.sh rename to manylinux2014_docker_images/install/install_golang.sh diff --git a/images/install/install_pip_packages.sh b/manylinux2014_docker_images/install/install_pip_packages.sh similarity index 100% rename from images/install/install_pip_packages.sh rename to manylinux2014_docker_images/install/install_pip_packages.sh diff --git a/images/install/install_proto3.sh b/manylinux2014_docker_images/install/install_proto3.sh similarity index 100% rename from images/install/install_proto3.sh rename to manylinux2014_docker_images/install/install_proto3.sh diff --git a/images/install/install_rpm_packages.sh b/manylinux2014_docker_images/install/install_rpm_packages.sh similarity index 100% rename from images/install/install_rpm_packages.sh rename to manylinux2014_docker_images/install/install_rpm_packages.sh diff --git a/ppc64le_builds/README.md b/ppc64le_builds/README.md new file mode 100644 index 0000000..d63a52e --- /dev/null +++ b/ppc64le_builds/README.md @@ -0,0 +1,19 @@ +# ppc64le Builds + +Dockerfiles and wheel build scripts for building TF on ppc64le. + +Maintainer: @wdirons (IBM) + +* * * + +`cpu/Dockerfile.manylinux_2014` extends from the +quay.io/pypa/manylinux2014_ppc64le docker image and installs java, bazel (from +source), and the minimal required pip packages to build TensorFlow. It also adds +the jenkins userid necessary to work with Oregon State University's Open Source +Lab jenkins environment. `gpu/Dockerfile.manylinux_2014.cuda10_1` extends from +the cpu image and adds CUDA 10.1 and cuDNN 7.6. + +The bash scripts are the equivalent of what is run in the TensorFlow builds done +with Jenkins (everything except the remote cache server). They can also be run +manually by cloning tensorflow and copying the script into the root of the +workspace. diff --git a/images/ppc64le/cpu/Dockerfile.manylinux_2014 b/ppc64le_builds/images/cpu/Dockerfile.manylinux_2014 similarity index 100% rename from images/ppc64le/cpu/Dockerfile.manylinux_2014 rename to ppc64le_builds/images/cpu/Dockerfile.manylinux_2014 diff --git a/images/ppc64le/gpu/Dockerfile.manylinux_2014.cuda10_1 b/ppc64le_builds/images/gpu/Dockerfile.manylinux_2014.cuda10_1 similarity index 100% rename from images/ppc64le/gpu/Dockerfile.manylinux_2014.cuda10_1 rename to ppc64le_builds/images/gpu/Dockerfile.manylinux_2014.cuda10_1 diff --git a/images/ppc64le/gpu/cuda.repo b/ppc64le_builds/images/gpu/cuda.repo similarity index 100% rename from images/ppc64le/gpu/cuda.repo rename to ppc64le_builds/images/gpu/cuda.repo diff --git a/ci_environments/ppc64le/cpu_build.sh b/ppc64le_builds/scripts/cpu_build.sh similarity index 100% rename from ci_environments/ppc64le/cpu_build.sh rename to ppc64le_builds/scripts/cpu_build.sh diff --git a/ci_environments/ppc64le/gpu_build.sh b/ppc64le_builds/scripts/gpu_build.sh similarity index 100% rename from ci_environments/ppc64le/gpu_build.sh rename to ppc64le_builds/scripts/gpu_build.sh diff --git a/tekton/README.md b/tekton/README.md new file mode 100644 index 0000000..b864c5d --- /dev/null +++ b/tekton/README.md @@ -0,0 +1,9 @@ +# Tekton CI + +perfinion's experimental directory for using Tekton CI with TensorFlow. + +Maintainer: @perfinion (SIG Build) + +* * * + +T.B.D. diff --git a/webhook/Dockerfile b/tekton/webhook/Dockerfile similarity index 100% rename from webhook/Dockerfile rename to tekton/webhook/Dockerfile diff --git a/webhook/app.py b/tekton/webhook/app.py similarity index 100% rename from webhook/app.py rename to tekton/webhook/app.py diff --git a/webhook/config.py b/tekton/webhook/config.py similarity index 100% rename from webhook/config.py rename to tekton/webhook/config.py diff --git a/webhook/requirements.txt b/tekton/webhook/requirements.txt similarity index 100% rename from webhook/requirements.txt rename to tekton/webhook/requirements.txt -- GitLab