Docs build improvements

[aaronmarkham]

Description

Docs building made easy... at least for Ubuntu.
Tested on a vanilla Ubuntu 16.04 instance.

Changes

• Fixes make docs - lein not found #11662 - adds Clojure docs build info
• Adds info on building R, Scala, or Doxygen docs manually
• Clarifies current Mac building situation
• Shows how to turn on or turn off building different API's docs
• MAKES DOCS BUILDING SUPER EASY!

Usage

• Clone mxnet or your fork on your Ubuntu instance
• Run the following to setup your instance's dependencies and build MXNet:

cd $MXNET_ROOT/docs/build_version_doc
./setup_docs_ubuntu.sh

Note: you might want to turn on web server redirects - the script outputs directions on how to do this.

• Run this after making updates or pulling the latest (be in $MXNET_ROOT directory):

make docs USE_OPENMP=1

• View or serve the docs generated in docs/_build/html/. You can create a deploy.sh script like this to make it easy for you to build new docs & serve:

make docs USE_OPENMP=1
sudo rm -rf /var/www/html
sudo cp -a docs/_build/html /var/www/

Note: if you run this on an existing server with old repo of mxnet laying around you might need to do the following from $MXNET_ROOT:

git fetch upstream
git rebase upstream/master
git submodule update --recursive
git submodule init
make clean

Then run the setup_docs_ubuntu.sh script.

[aaronmarkham]

@kpmurali @larroy @thomelane @ThomasDelteil @marcoabreu @nswamy - you guys will probably appreciate this. LMK if I can make any improvements, or if you find any bugs.

[marcoabreu]

That's awesome, thanks a ton!

I've been thinking about whether we could make a new Dockerfile for the doc generation (last stage) which contains nothing else than ubuntu vanilla and then runs your setup as well as build script. This way, we don't have to maintain two solutions and we got full test coverage. We would just need to modify your script a bit so we can skip compiling MXNet as part of the setup process.

Btw, why do we need USE_OPENMP=1 for make docs?

Are you going to provide a one-click script to automate the steps you explained here? Also, is this documented somewhere else besides the description of this PR?

[aaronmarkham]

I updated the README and it uses this script: https://github.com/apache/incubator-mxnet/pull/11670/files#diff-3bb49ab660a5d5d6192626615a92a775R128

I'd love to separate the docs/website build to a different pipeline. I think there's a lot of money spent doing CI on text updates that don't need all of that testing.

Regarding OPENMP, somewhere along the line I saw something that made me believe the build runs faster with it turned on. shrug Would love to know one way or the other and get rid of it if we can, or maybe have the make docs trigger use the right flags by default.

For one-click: it sort of is. You need the repo, then you run the script. That's the setup. Then, if you're developing with the website or the docs, you need to run make docs all of the time... That's why I suggest the deploy.sh script that rolls up the make docs step with moving the files to the web server directory. That's what I use. I can add that script too - I just thought it might be too trivial.

[larroy]

Thanks @aaronmarkham I personally would prefer to have this streamlined in a docker, but I guess this makes it closer to that. I wonder why we have to rebuild the project to generate docs. Is it because operator generation?

[aaronmarkham]

@larroy - It does seem to skip building the full project if no change is detected, but ya, as soon as some op changes it triggers a full rebuild. I have some experience making doxygen and API docs build not so monolithic with cmake. Did we ever decide to switch to that?

As for Docker, we're pretty close to that now, but since I like to build and serve for preview links on a server that doesn't go offline all the time (like my mac), I don't find a Docker option all that useful for me. What's your use case?

[larroy]

Yes, we have decided to switch to Cmake and there's people working on that (@lebeg) but it will take a while. If possible is better to start new things in CMake

[lebeg]

Yes, cmake would be the preferable way

[aaronmarkham]

I just found out I put the wrong script name in the description above. I had moved the logic to ./setup_docs_ubuntu.sh, so just run that when you try this.

Krishnan found the R gbg step to fail on his instance, so I'm pointing that to the CI's copy of the file like I did with sbt.

[aaronmarkham]

@szha @marcoabreu @nswamy - can any of you merge? Any reason to keep this in limbo?

[aaronmarkham]