Automatically generate nb-javac from JDK sources

.github/workflows/main.yml

@@ -1,45 +1,25 @@
 name: CI
-
+#
 # Controls when the action will run. Triggers the workflow on push or pull request
 # events but only for the master branch
-on:
-  push:
-    branches: [ master ]
+on: [push]
+#  push:
+#    branches: [ master ]
 jobs: 
   linux: 
-    name: "Linux Build"
+    name: "Backported nb-javac"
     runs-on: ubuntu-latest
-    steps: 
-      - 
-        uses: actions/checkout@v2
-      - 
-        name: "Build nb-javac"
-        run: "ant -f ./make/langtools/netbeans/nb-javac clean jar"
-      -  
-        name: "Unit Test nb-javac"
-        run: ant -f ./make/langtools/netbeans/nb-javac -Dnb.internal.action.name=test -Dignore.failing.tests=true test
-
-  macos: 
-    name: "MacOS build"
-    runs-on: macos-latest
-    steps: 
-      - 
-        name: Checkout
-        uses: actions/checkout@v2
-      - 
-        name: "Build nb-javac"
-        run: "ant -f ./make/langtools/netbeans/nb-javac clean jar"  
-      -  
-        name: "Unit Test nb-javac"
-        run: ant -f ./make/langtools/netbeans/nb-javac -Dnb.internal.action.name=test -Dignore.failing.tests=true test
-
-  windows: 
-    name: "Windows build"
-    runs-on: windows-latest
-    steps: 
-      - 
-        name: Checkout
-        uses: actions/checkout@v2
-      - 
-        name: "Build nb-javac"
-        run: "ant -f ./make/langtools/netbeans/nb-javac clean jar"
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up JDK 8
+        uses: actions/setup-java@v1.4.3
+        with:
+            java-version: 8
+      - name: Record JDK8
+        run: echo "$JAVA_HOME" | tee jdk8
+      - name: Set up JDK 14
+        uses: actions/setup-java@v1.4.3
+        with:
+            java-version: 14
+      - name: Do build
+        run: JAVA_HOME8=`cat jdk8` JAVA_HOME14=$JAVA_HOME bash build.sh

.gitignore

@@ -1,5 +1,9 @@
 /build/
 /dist/
+/jdk/
+/lib/
+/src/
+/jackpot.jar
 /.idea/
 /.vscode/
 nbproject/private/
@@ -14,6 +18,7 @@ test/nashorn/lib
 NashornProfile.txt
 **/JTreport/**
 **/JTwork/**
+**/compiler.jar
 /make/langtools/netbeans/nb-javac/build/
 /make/langtools/netbeans/nb-javac/dist/
 /make/langtools/netbeans/nb-javac/nbproject/private/

BUILD.md

@@ -0,0 +1,95 @@
+# Hacking Guide for the automatically generated [nbjavac](README.md)
+
+The idea of the new build system is to take the JDK 16+ `javac` sources and
+automatically convert them to run on JDK 8+. As a result the sources come 
+from real JDK repository. The `nbjavac` repository doesn't contain them. 
+This repository only contains the build scripts and
+description of [advanced refactorings](https://netbeans.apache.org/jackpot/HintsFileFormat.html).
+Use:
+
+```bash
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac jar
+```
+
+to build everything at once. Read below to control individual steps of the build.
+
+
+### Getting the JDK repository
+
+The build requires JDK repository in `jdk` subdirectory of the root of `nb-javac` repository.
+If such directory doesn't exist, the build checks out one:
+
+```bash
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac init \
+    -Djdk.git.url=https://github.com/openjdk/jdk16 \
+    -Djdk.git.commit=jdk-16+36
+```
+
+If the `jdk` directory is present the build leaves its content untouched. E.g.
+a developer may clone the `jdk` repository manually, switch its content to any other tag,
+make changes in the `jdk/src/java.compiler/` or `jdk/src/jdk.compiler/` directories,
+etc. Bugfixes, features and other changes to `javac` sources are supposed to be done 
+in the `jdk` subdirectory and integrated into the JDK's `javac` official repository.
+
+One can discard any changes by `rm -rf jdk`. Then the subsequent build checks
+a fresh copy of the `jdk` repository from scratch. The default values for
+`jdk.git.url` and `jdk.git.commit` properties are in the
+`./make/langtools/netbeans/nb-javac/nbproject/project.properties`
+file.
+
+
+### Automatically processing the sources
+
+Once the JDK's `javac` sources are in the `jdk` subdirectory, it is necessary
+to apply [advanced refactorings](./make/langtools/netbeans/nb-javac/src/META-INF/upgrade/nbjavac.hint)
+to them. This is done by executing the [jackpot](https://netbeans.apache.org/jackpot/HintsFileFormat.html)
+target:
+
+```bash
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac jackpot
+```
+
+This step copies the `javac` sources from the `jdk` subdirectory into a sibling
+`src` subdirectory and applies necessary transformations to them.
+The goal of such transformations is to eliminate usage of JDK9+ APIs
+and replace them with JDK8 only APIs.
+
+The sources under the `src/java.compiler` and `src/jdk.compiler` shall not
+be edited manually. Rather than that edit the sources in the original
+`jdk/src/java.compiler/` and `jdk/src/jdk.compiler/` directories. To apply
+the refactorings again execute:
+
+```bash
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac clean jackpot
+```
+
+### The build
+
+As described in [general documentation](README.md) use the following command to
+generate the final JAR files:
+
+```bash
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac clean jar
+```
+
+JARs `nb-javac-*-api.jar` and `nb-javac-*-impl.jar` are going to appear
+at location `./make/langtools/netbeans/nb-javac/dist/`.
+
+### Debug & Develop 
+
+Open the `nb-javac` project in NetBeans IDE with
+
+```bash
+$ netbeans --open make/langtools/netbeans/nb-javac/
+```
+
+and you should be able to debug a test (for example `StringWrapperTest`) with following command line:
+
+```bash
+$ JAVA_HOME=/jdk-8/ ant -f make/langtools/netbeans/nb-javac test \
+    -Dincludes=**/StringWrapperTest* \
+    -Ddebug.jvmargs=-agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=y
+```
+
+Connect the NetBeans IDE to port 5005 and step through the `nb-javac`
+generated code.

README.md

@@ -1,38 +1,52 @@
 # About nb-javac!
-"nb-javac" is a patched version of OpenJDK "javac", i.e., the Java compiler. This has long been part of NetBeans, providing a highly tuned Java compiler specifically for the Java editor i.e., parsing and lexing for features such as syntax coloring, code completion.
+_nb-javac_ is a backport of OpenJDK "javac", i.e., the Java compiler. _nbjavac_
+takes sources from the latest JDK and backports them to run on JDK8+.
+The _nb-javac_ has long been part of NetBeans, providing a highly tuned Java compiler
+specifically for the Java editor i.e., parsing and lexing for features
+such as syntax coloring, code completion. 
 
 # Prerequisite
   - Git
   - Ant 1.9.9 or above
-  - JDK 8 or above (to build nb-javac)
-  - Apache Maven
+  - JDK 14 to build
+  - JDK 8 to test
+  - Apache Maven to publish to Maven central
 
 # Building nb-javac jar files
-1. Obtain the code with the following command
+
+Detailed description of the [build process](BUILD.md) is available in a
+separate [development documentation](BUILD.md). Here are just
+the most straigtforward steps to get the final artifacts.
+
+### Obtain the code with the following command
 
 ```
 $ git clone https://github.com/oracle/nb-javac.git
 ```
 
-2. To get a specific version use the following command
+### Get a specific version use the following command
 
 ```bash
 $ git checkout <release_tag_name> 
 ```
 
-3. Run the below command to build nb-javac.
+### Run the below command to build nb-javac.
 
 ```bash
-$ ant -f ./make/langtools/netbeans/nb-javac clean jar
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac clean jar
 ```
 
-Two jars namely `javac-api*.jar` and `javac-impl*.jar` are going to appear
-at location `./make/langtools/netbeans/nb-javac/dist/`
+Two jars namely `nb-javac-*-api.jar` and `nb-javac-*-impl.jar` are going to appear
+at location `./make/langtools/netbeans/nb-javac/dist/`. It is also possible to
+sanity test the generated Javac on JDK8:
+```bash
+$ JAVA_HOME=/jdk-8/ ant -f ./make/langtools/netbeans/nb-javac test
+```
 
-4. Run below command to zip the source code of nb-javac
+### Generate ZIP with the source code of nb-javac
 
 ```bash
-$ ant -f ./make/langtools/netbeans/nb-javac zip-nb-javac-sources
+$ JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac zip-nb-javac-sources
 ```
 
 # Publishing to maven central / OSSRH
@@ -45,13 +59,13 @@ $ ant -f ./make/langtools/netbeans/nb-javac zip-nb-javac-sources
 
 3. Run
    ```
-   ant -f ./make/langtools/netbeans/nb-javac publish-to-ossrh-snapshots -Dmaven.groupId=your.grp.id
+   JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac publish-to-ossrh-snapshots -Dmaven.groupId=your.grp.id
    ```
    to publish snapshot artifacts (https://oss.sonatype.org/content/repositories/snapshots/)
 
 4. Run
    ```
-   ant -f ./make/langtools/netbeans/nb-javac publish-to-maven-central -Dmaven.groupId=your.grp.id
+   JAVA_HOME=/jdk-14/ ant -f ./make/langtools/netbeans/nb-javac publish-to-maven-central -Dmaven.groupId=your.grp.id
    ```
    to stage the release, which will get promoted to maven central, after it has
    been manually released.

StripModuleHashes.java

@@ -0,0 +1,49 @@
+import com.sun.tools.classfile.Attribute;
+import com.sun.tools.classfile.Attributes;
+import com.sun.tools.classfile.ClassFile;
+import com.sun.tools.classfile.ClassWriter;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashMap;
+import java.util.jar.JarInputStream;
+import java.util.jar.JarOutputStream;
+import java.util.zip.ZipEntry;
+
+public class StripModuleHashes {
+
+    public static void main(String[] args) throws Exception {
+        try (InputStream in = new FileInputStream(args[0]);
+             OutputStream out = new FileOutputStream(args[1])) {
+            out.write(in.read());//J
+            out.write(in.read());//M
+            out.write(in.read());//major version
+            out.write(in.read());//minor version
+            try (JarInputStream jis = new JarInputStream(in);
+                 JarOutputStream jos = new JarOutputStream(out)) {
+                ZipEntry e;
+                while ((e = jis.getNextEntry()) != null) {
+                    jos.putNextEntry(e);
+
+                    if (e.getName().endsWith("/module-info.class")) {
+                        ClassFile cf = ClassFile.read(jis);
+                        Attributes attrs = cf.attributes;
+                        HashMap<String, Attribute> newAttrs = new HashMap<>(attrs.map);
+                        newAttrs.remove(Attribute.ModuleHashes);
+                        ClassFile newCF = new ClassFile(cf.magic, cf.minor_version, cf.major_version, cf.constant_pool, cf.access_flags, cf.this_class, cf.super_class, cf.interfaces, cf.fields, cf.methods, new Attributes(newAttrs));
+                        new ClassWriter().write(newCF, jos);
+                    } else {
+                        int read;
+
+                        while ((read = jis.read()) != (-1)) {
+                            jos.write(read);
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+}

build-jdk.sh

@@ -0,0 +1,42 @@
+if ! [ -f $JAVA_HOME16/bin/javac ]; then
+  echo Specify JAVA_HOME16 environment variable!
+  exit 1
+fi
+
+rm -rf build/test
+mkdir -p build/test
+mkdir -p build/test/modules
+cp $JAVA_HOME16/jmods/* build/test/modules
+#clear ModuleHashes on java.base:
+$JAVA_HOME16/bin/java --add-modules jdk.jdeps --add-exports jdk.jdeps/com.sun.tools.classfile=ALL-UNNAMED StripModuleHashes.java $JAVA_HOME16/jmods/java.base.jmod build/test/modules/java.base.jmod
+rm build/test/modules/java.compiler.jmod
+rm build/test/modules/jdk.compiler.jmod
+mkdir -p build/test/src/java.compiler
+cp src/java.compiler/share/classes/module-info.java build/test/src/java.compiler
+
+patch -R build/test/src/java.compiler/module-info.java temporary-patches/test-java.compiler
+
+mkdir -p build/test/src/jdk.compiler
+cp src/jdk.compiler/share/classes/module-info.java build/test/src/jdk.compiler
+
+patch build/test/src/jdk.compiler/module-info.java temporary-patches/test-jdk.compiler
+
+mkdir -p build/test/out/java.compiler
+cp -r make/langtools/netbeans/nb-javac/build/classes/javax build/test/out/java.compiler/
+cp -r make/langtools/netbeans/nb-javac/build/classes/nbjavac build/test/out/java.compiler/
+
+mkdir -p build/test/out/jdk.compiler
+cp -r make/langtools/netbeans/nb-javac/build/classes/com build/test/out/jdk.compiler/
+cp -r make/langtools/netbeans/nb-javac/build/classes/jdk build/test/out/jdk.compiler/
+
+$JAVA_HOME16/bin/javac --module-source-path build/test/src/ -d build/test/out `find build/test/src/ -type f -name "*.java"`
+
+$JAVA_HOME16/bin/jmod create --class-path build/test/out/java.compiler/ build/test/modules/java.compiler.jmod
+
+mkdir -p build/test/expanded
+
+$JAVA_HOME16/bin/jmod extract --dir=build/test/expanded $JAVA_HOME16/jmods/jdk.compiler.jmod
+
+$JAVA_HOME16/bin/jmod create --class-path build/test/out/jdk.compiler/ --cmds build/test/expanded/bin/ --legal-notice build/test/expanded/legal/ --libs build/test/expanded/lib/ --man-pages build/test/expanded/man/ --module-version 16 build/test/modules/jdk.compiler.jmod
+
+$JAVA_HOME16/bin/jlink -p build/test/modules --add-modules ALL-MODULE-PATH --output build/test/jdk

build.sh

@@ -0,0 +1,29 @@
+set -x
+set -e
+
+if ! [ -f $JAVA_HOME8/bin/javac ]; then
+  echo Specify JAVA_HOME8 environment variable!
+  exit 1
+fi
+
+if ! [ -f $JAVA_HOME8/jre/lib/rt.jar ]; then
+  echo Specify JAVA_HOME8 environment variable!
+  exit 1
+fi
+
+if ! [ -f $JAVA_HOME14/bin/javac ]; then
+  echo Specify JAVA_HOME14 environment variable!
+  exit 2
+fi
+
+if ! [ -f $JAVA_HOME14/jmods/java.base.jmod ]; then
+  echo Specify JAVA_HOME14 environment variable!
+  exit 2
+fi
+
+export JAVA_HOME=$JAVA_HOME14
+ant $ANT_ARGS_EXTRA -f make/langtools/netbeans/nb-javac clean jar
+
+
+export JAVA_HOME=$JAVA_HOME8
+ant $ANT_ARGS_EXTRA -f make/langtools/netbeans/nb-javac test