I also wrote an article explaining how I achieved this. It’s been published on the spring.io blog.

The article is split into two parts. The first section provides an overview and shows the use of Function Calling. The second part is all about Retrieval-Augmented Generation (RAG).

Links below. Happy reading!

AI Meets Spring Petclinic: Implementing an AI Assistant with Spring AI (Part I)

AI Meets Spring Petclinic: Implementing an AI Assistant with Spring AI (Part II)

The event had a host of distinguished guests from 7 countries.

Although over 600 registered, we had to limit the attendees to only 250 due to the venue's capacity. Luckily, we had everything recorded for you to enjoy!

Here is the playlist with all talks:

Spring Framework founder and project lead Juergen Hoeller presented Spring Framework 6.0, which was released just in time for the event:

DaShaun Carter presented Spring Boot 3:

Cora Iberkleid presented Spring Cloud Gateway:

Spring Boot and Spring Cloud founder David Syer discussed how to run untrusted code using WebAssembley:

Yours truly presented a talk on production considerations for Spring on Kubernetes:

Oleg Šelajev and Cora Iberkleid gave an introduction to TestContainers:

Director of Dev and Infrastructure at Israel's Social Security, Yuri Fux, presented his team's modernization journey to Spring Boot and Spring Cloud. This was the only session that is in Hebrew:

Olga Maciaszek-Sharma discussed declarative HTTP clients in Spring:

Lastly, I presented along with Scott Rosenberg an end-to-end demo using Spring Boot and Tanzu Application Platform:

Overall, this was truly a fun event to organize and attend. The venue at the Peres center of innovation was beautiful, and the speakers were the best there is.

We're planning to have a Spring One Tour in 2023 as well. Stay tuned!

Tell me a joke

I started out with just playing around and trying out a few things like asking it to tell jokes in certain themes. This one was just hilarious:

I then asked it to create a new plot for a Friends episode:

Finally, I asked him to create a movie plot for Back to the future part 4. I would love for someone to make this!

Let's get geeky

Alright. This is all cute but let's get to business.

First, I asked him to explain Kubernetes, but for different audiences: A 5-years old, an elderly person, and a computer PHD:

This one truly showed me how ChatGPT understands context. This is so impressive to be able to explain a complex subject to different audiences.

Next, I asked ChatGPT to create a lullaby on Horizontal Pod Autoscalers. It did a pretty good job:

Lullabies are cool, but can it rap? Yes it can.

I then wanted to test some assertions I have about best practices. It was detailed enough to get me rethinking some things:

I think the most impressive part was getting responses for very obscure edge cases. For example - what's the recommended way to move a BOSH deployment from a single availability zone to multiple availability zones?

Or even, what exactly gets backed up when using BOSH backup and restore?

Let's get coding

Ok, the interactive conversation is fun, but does this thing know how to code?

Why yes, yes it does.

Here it is creating a simple Spring Boot application that just works, along with a very nice explanation on the implementation:

Here it is providing a working example for RabbitMQ streams:

Putting it all together

Ok. Let's use ChatGPT as my co-pilot assistant when developing a new application.

The application takes orders for T-Shirts, written in React for the frontend and Spring for the backend. Spring writes to a Postgres database. I plan to deploy the application to a Kubernetes cluster using Knative serving, and integrate it with Github OIDC. Can it be done? It can get you 85% there in less then 10 minutes.

Summary

I truly believe ChatGPT is going to change the way we work. It feels a lot like Altavista did in the early 2000s when Google started taking over. It's just leaps and bounds better. I'm sure some teams at Google are planning to come up with good alternatives as we speak.

ChatGPT isn't perfect though. I already found a few examples it provided that were simply wrong. It is still AI, and it is only as good as the data sets it is modeling. If your source data is bad, your answers would be as well.

But these are very early days. It will get better, and possibly very quickly.

Fun times ahead.

The 2021 State of Spring report has two main data points:

• Spring is the defacto standard for running cloud-native applications in Java.
• Kubernetes is the defacto standard for running containerized workloads in production.

This begs the question - what kind of best practices and considerations should be taken into account when running Spring on Kubernetes?

In this article, I shall highlight some of the learnings and best practices that have formed around running Spring (and more broadly Java) applications on Kubernetes. This document is not entirely based on first-hand experiences, but more a collection of conclusions and best practices I identified in the community and when working with customers. These are also based on my own personal observations and are not endorsed by my employer.

You might think that in order to run a Spring Boot app in Kubernetes, all you have to do is quickly jot down a Dockerfile and be done with it.

As Kramer learned the hard way, it's never that easy:

You may consider a developer platform such as Tanzu Application Platform to offload some of these infrastructure decisions to a platform, but even with a PaaS there are architecture and topology decisions to be made. Besides, if you build it - you own it. It's probably a good idea to know what you're owning.

Build & Run

There two separate yet equally important aspects of running a production-ready Spring application in Kubernetes:

• Building the OCI image (a.k.a., the "Docker" image)
• Running the OCI image on Kubernetes

Each step has its own gotchas and intricacies. Let's review these in details.

Building the image

At first, this sounds fairly simple. Just Google some sample Dockerfile for a Java application and use it as reference.

In reality, there are many important decisions that will impact your target image and how it behaves with regards to the Java Virtual Machine it is running.

Choose a Linux distribution

There are so many Linux distros out there that it gets hard to keep up. Should you go with Ubuntu? RedHat Enterprise Linux? Debian? Suse? Alpine? Distroless? Which one makes more sense for your JDK-based OCI image?

For containerized JDK workloads, the question should actually be more generalized. It should be focused on which Linux kernel you are going to base your container on.

There's a good reason why you can run a Red Hat Enterprise Linux container on an Ubuntu VM, or a Debian Linux container on a PhotonOS VM. The important pieces for a Linux-based container are:

• libc, which is at the core of OCI images. This is practically the same programming interface, regardless of the Linux distribution you plan to use.
• The Linux kernel Application Binary Interface (ABI), which guarantees backwards compatibility between different Linux kernel versions.

Based on this information, it sounds like you shouldn't really care which Linux distribution you'll use. However, the devil is in the details.

There are two popular implementations for the libc interface:

• GNU libc: This is the standard library that you probably use every day. It is used by Ubuntu, Debian, CentOS, RHEL, SUSE and more. Its downside is that it is a fairly large and heavy codebase (in container terms).
• Musl libc: This is a newer implementation of the library. Used by Alpine Linux, it is much smaller in size compared to GNU libc and is meant to be lightweight, fast and simple. However, there are caveats. Musl libc actually has functional differences compared to GNU libc - things like regular expressions, EOF and multithreading could behave differently based on the implementation. [See this detailed comparison]

It sounds like the implementation details are less important when you work against an interface, but that's not the case for the Java Virtual Machine. Until fairly recent versions, the JVM relied on features that were only available in GNU libc, and modifications had to be made to support the Musl implementation.

Starting with Java 16, both implementations are supported. Since you'll probably want to stick to Java long-term support releases, in practice this means that you should only rely on Alpine-based distros starting with Java 17 and later.

For best compatibility, use a GNU-based libc Linux implementation. At VMware Tanzu, we use Ubuntu extensively for Spring-based images and find it to be an excellent choice, but other solutions should work just as well: RHEL, Debian, PhotonOS, SUSE, CentOS etc.

Choose a JDK distribution

If you ever installed SDKman on your machine, you know there are a plethora of JDK distros out there:

If that's not enough, there is also a long list of JDK base images to choose from in Docker Hub. Merely searching for Java returns a single official image with the label DEPRECATED; use openjdk or other jdk implementations instead.

A search for openjdk returns over 6,500 results. The first two results are "Docker Official images", one for OpenJDK with over a billion downloads, and one for Eclipse Temurin with over 10 million downloads:

These sounds like good choices. Just be aware that "official" doesn't necessarily mean what you think it means. It just means that the build itself is done by Docker - and sometimes this can cause more harm than good.

Matt Raible has a very good article on the different distros available out there, although some distribution names have changed since its publishing at 2019.

For OCI images, there are three variants that you should consider:

• Oracle JDK - requires license for production use. This is the "official" JDK.
• OpenJDK, built and maintained by Oracle, and free. Oracle's OpenJDK doesn't offer support for Java 8 anymore.
• Adoptium (the rebranded AdoptOpenJDK distribution) is built and maintained by the community. It is compatible with Oracle's version while having more "friendly" licensing terms. Vendors such as Bellsoft, IBM, Amazon and others offer JDK distributions based on Adoptium.

The decision on which JDK to use has many factors, so it's hard to make a single recommendation.

• If your company already has an Oracle license, use their JDK base images.
• If you are running on a single public cloud provider, consider using that cloud provider's JDK release, such as Amazon Corretto or Microsoft's build of OpenJDK.
• If you are looking for a fair and production-ready distribution that is community-driven and can run anywhere, use the Adoptium-provided base images, or one of the various flavors based off of it with commercial support, such as Azul or Bellsoft Liberica. VMware Tanzu (the business unit that the Spring team is a part of) uses Bellsoft Liberia in products such as Tanzu Build Service and Tanzu Application Platform.

Use the Adoptium base images such as eclipse-temurin:17-jre-jammy, or a supported flavor of the builds. If you run on a single public cloud, consider using that cloud provider's base image. If you have a license for the Oracle JDK, use it.

Choose a JDK version

Oh, this one's easy. Use the latest LTS version that you can. At the time of this writing, the latest LTS version is Java 17.

LTS stands for Long-Term Support, and in general the industry tries to stick to those releases unless there are very important reasons to move to other versions. Non-LTS versions are released every 6 months and are supported for fairly short cycles, while LTS versions are being supported for many years (5+ based on the distribution).

Starting with Spring Framework 6.0, the minimum supported version will be JDK 17.

Support is just one aspect of the selected version. Each version also introduces new features, but more importantly - better performance, better security and improvements to the garbage collector.

Here's the startup time for the famous Spring Boot Petclinic application while running in a Java 11-based container based on Bellsoft Liberica on a 4-cores machine running macOS:

And here's the same application running on a Java 17 container, also based on Bellsoft Liberica, on the same 4-cores machine running macOS:

That's a 1.5 seconds difference just by upgrading your JVM. It may not sound like a lot, but in a Kubernetes environment where pods go up and down at any given moment, that's a lot.

If you can't use Java 17 due to compatibility issues, use Java 11 - the previous LTS version.

Java 8

Now, I know what some of you are thinking right about now: "What are you talking about? we're still stuck with Java 8 and we can't upgrade anytime soon". If that's you - you're not alone. According to a recent survey by JRebel, more than a third of Java developers are still using Java 8 in production, and that's not going to change in the near future. If you still have to use Java 8, that's ok (🫤), but start planning your strategy to upgrade. Also - there are several important caveats to consider:

In containerized environments, Use Java 8 update 191 or later. Older versions of Java 8 are not compatible with containerized environments, and the JVM mistakenly reports the host's CPU and memory instead of the container's. Here's an example for running an application with Java 8 update 131 on a Linux VM with 4 CPUs and 16 GB of RAM:

As you can see, in Java 8 update 131 - the container uses all available CPUs on the host although I asked the container to limit the container to just 2 CPUs. Also - the available memory is set to 4 GB although I asked for a container with just 256 MB (the max heap size is always calculated as a portion of machine's available memory, as a default that it is 1/4 of the host's memory). This can cause extremely unpredictable results, especially since many thread pools such as those used by Tomcat and thread pools use the number of available processors (Runtime.getAvailableProcessors()) to calculate the size of the pool, which translates to larger native memory usage, which eats more of the total available memory, which in turn, again, can cause havoc.

If that's not enough, the JVM also chooses the garbage collector based on available CPU and Memory. You will get different garbage collectors based on different CPU and memory configurations.

So, if you have to use Java 8 - use a recent version, update 191 or later. You'll also get a ton of security patches which are critical for production environments.

Here's the same docker run command on the same Linux VM, but with Java 8 update 292:

This time, our JVM reports a max heap size of 132 MB and no parallel garbage collection threads. Also - all thread pool using the Runtime.getAvailableProcessors() command will get the correct number of processors, and therefore will generate reasonable pools with reasonable native memory sizes.

An important note that goes without saying: there are always refinements and enhancements, and bugs are always a possibility, so use the latest patch version based on your target JDK version. Just recently, a patch was made to support cgroups v2 because the first time the feature got implemented, only cgroups v1 was available. Adding support for cgroups v2 with Java 8 is still ongoing development at the time of this writing.

Use Java 17 LTS. If you can't use Java 17, use Java 11 LTS. If you can't use Java 11, use Java 8 update 191 or later. Do not use earlier versions of Java in a containerized environment such as Kubernetes.

Writing a Dockerfile

If you'll Google "Spring Boot Dockerfile", you might run into this article. In it, the first section lists the following example:

FROM eclipse-temurin:17-jre-jammy
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

This Dockerfile will work just fine, but I would advise you to avoid it. The above Dockerfile places the Spring Boot "fat jar" into a single OCI layer. This uber jar can easily reach 150 MB or more. What does this mean in practice? It means that for every new build, even for the simplest code change in our application:

• A new 150 MB layer (read: tarball) will be created
• The layer will be uploaded to your OCI registry
• When Kubernetes pulls the latest image to the node that runs the container, the entire 150 MB layer will need to be pulled.

This will happen on every single code change and rebuild of the image. In reality, only a very small sliver of compiled code has changed. Dockerfile treats each new line as a new layer, so it would make a lot more sense to put the third party dependencies in their own layer and have our custom code in its own layer. But how should you layer your Spring Boot image? Which app stack should be treated as its own layer?

Luckily, Spring Boot has you covered. Let's take the same Petclinic application, build it, and ask Spring to list what it thinks are the correct layers that are suitable for an OCI image. We can use a special execution mode called layertools:

$ ls -l target
total 105992
-rw-r--r--   1 odedia  staff  53169785 Sep  5 11:10 spring-petclinic-2.7.0-SNAPSHOT.jar
-rw-r--r--   1 odedia  staff    391665 Sep  5 11:10 spring-petclinic-2.7.0-SNAPSHOT.jar.original
drwxr-xr-x  22 odedia  staff       704 Sep  5 11:10 surefire-reports
drwxr-xr-x   3 odedia  staff        96 Sep  5 11:09 test-classes

$ java -Djarmode=layertools -jar target/spring-petclinic-2.7.0-SNAPSHOT.jar list
dependencies
spring-boot-loader
snapshot-dependencies
application

As you can see, instead of running the Spring Boot application, passing the argument -Djarmode=layertools list to the java command will simply list the layers that make sense for Spring Boot application. In our case, these are:

• Dependencies - the Spring Boot and other frameworks' release-based dependencies. These only change when we upgrade a Spring Boot version or the third party framework version.
• Spring Boot Loader - this is the code that loads our Spring Boot app into the JVM to manage the bean lifecycle, among other things. This also rarely changes.
• Snapshot Dependencies - these dependencies are changing much more often. It's possible that on each new build it would require us to pull latest snapshot. As a result, this layer is the closest to our application code.
• Application - This is our application code from src/main/java (in the case of maven).

So, Spring Boot helps us with hints on how best to layer our application. But wouldn't it be better if it could actually prepare the layers for us? Well, it can do exactly that:

$ java -Djarmode=layertools -jar target/spring-petclinic-2.7.0-SNAPSHOT.jar extract --destination target/tmp

$ ls -l target/tmp
total 0
drwxr-xr-x  4 odedia  staff  128 Sep  5 11:17 application
drwxr-xr-x  3 odedia  staff   96 Sep  5 11:17 dependencies
drwxr-xr-x  2 odedia  staff   64 Sep  5 11:17 snapshot-dependencies
drwxr-xr-x  3 odedia  staff   96 Sep  5 11:17 spring-boot-loader

Let's see the contents of the dependencies folder for example:

$ ls target/tmp/dependencies
BOOT-INF
$ ls target/tmp/dependencies/BOOT-INF
lib
$ ls target/tmp/dependencies/BOOT-INF/lib
HdrHistogram-2.1.12.jar                       jakarta.activation-1.2.2.jar                  spring-boot-actuator-autoconfigure-2.7.1.jar
HikariCP-4.0.3.jar                            jakarta.activation-api-1.2.2.jar              spring-boot-autoconfigure-2.7.1.jar
LatencyUtils-2.0.3.jar                        jakarta.annotation-api-1.3.5.jar              spring-boot-jarmode-layertools-2.7.1.jar
antlr-2.7.7.jar                               jakarta.persistence-api-2.2.3.jar             spring-context-5.3.21.jar
aspectjweaver-1.9.7.jar                       jakarta.transaction-api-1.3.3.jar             spring-context-support-5.3.21.jar
attoparser-2.0.5.RELEASE.jar                  jakarta.validation-api-2.0.2.jar              spring-core-5.3.21.jar
bootstrap-5.1.3.jar                           jakarta.xml.bind-api-2.3.3.jar                spring-data-commons-2.7.1.jar
byte-buddy-1.12.11.jar                        jandex-2.4.2.Final.jar                        spring-data-jpa-2.7.1.jar
cache-api-1.1.1.jar                           jaxb-runtime-2.3.6.jar                        spring-expression-5.3.21.jar
checker-qual-3.5.0.jar                        jboss-logging-3.4.3.Final.jar                 spring-jcl-5.3.21.jar
classgraph-4.8.139.jar                        jul-to-slf4j-1.7.36.jar                       spring-jdbc-5.3.21.jar
classmate-1.5.1.jar                           log4j-api-2.17.2.jar                          spring-orm-5.3.21.jar
ehcache-3.10.0.jar                            log4j-to-slf4j-2.17.2.jar                     spring-tx-5.3.21.jar
font-awesome-4.7.0.jar                        logback-classic-1.2.11.jar                    spring-web-5.3.21.jar
h2-2.1.214.jar                                logback-core-1.2.11.jar                       spring-webmvc-5.3.21.jar
hibernate-commons-annotations-5.1.2.Final.jar micrometer-core-1.9.1.jar                     thymeleaf-3.0.15.RELEASE.jar
hibernate-core-5.6.9.Final.jar                mysql-connector-java-8.0.29.jar               thymeleaf-extras-java8time-3.0.4.RELEASE.jar
hibernate-validator-6.2.3.Final.jar           postgresql-42.3.6.jar                         thymeleaf-spring5-3.0.15.RELEASE.jar
istack-commons-runtime-3.0.12.jar             slf4j-api-1.7.36.jar                          tomcat-embed-core-9.0.64.jar
jackson-annotations-2.13.3.jar                snakeyaml-1.30.jar                            tomcat-embed-el-9.0.64.jar
jackson-core-2.13.3.jar                       spring-aop-5.3.21.jar                         tomcat-embed-websocket-9.0.64.jar
jackson-databind-2.13.3.jar                   spring-aspects-5.3.21.jar                     txw2-2.3.6.jar
jackson-datatype-jdk8-2.13.3.jar              spring-beans-5.3.21.jar                       unbescape-1.1.6.RELEASE.jar
jackson-datatype-jsr310-2.13.3.jar            spring-boot-2.7.1.jar                         webjars-locator-core-0.50.jar
jackson-module-parameter-names-2.13.3.jar     spring-boot-actuator-2.7.1.jar
$

And here is our application's compiled code:

$ ls target/tmp/application/BOOT-INF/classes
application-mysql.properties    application.properties          db                              messages                        static
application-postgres.properties banner.txt                      git.properties                  org                             templates

As you can see - this process takes our uber jar and explodes it to make it suitable for a layered OCI image. If you've been around long enough like me and have flashbacks to exploded EAR files, you're not alone. Technology has a tendency to repeat itself.

We can now incorporate the exploded directories into our Dockerfile:

FROM eclipse-temurin:17-jre-jammy as builder
WORKDIR application
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM eclipse-temurin:17-jre-jammy
WORKDIR application
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

Let's see our OCI image in action:

$ # With layered image approach

$ docker run -p 8080:8080 petclinic


              |\      _,,,--,,_
             /,`.-'`'   ._  \-;;,_
  _______ __|,4-  ) )_   .;.(__`'-'__     ___ __    _ ___ _______
 |       | '---''(_/._)-'(_\_)   |   |   |   |  |  | |   |       |
 |    _  |    ___|_     _|       |   |   |   |   |_| |   |       | __ _ _
 |   |_| |   |___  |   | |       |   |   |   |       |   |       | \ \ \ \
 |    ___|    ___| |   | |      _|   |___|   |  _    |   |      _|  \ \ \ \
 |   |   |   |___  |   | |     |_|       |   | | |   |   |     |_    ) ) ) )
 |___|   |_______| |___| |_______|_______|___|_|  |__|___|_______|  / / / /
 ==================================================================/_/_/_/

:: Built with Spring Boot :: 2.7.1


2022-09-05 08:36:35.170  INFO 1 --- [           main] o.s.s.petclinic.PetClinicApplication     : Starting PetClinicApplication v2.7.0-SNAPSHOT using Java 17.0.4.1 on 73cac62e34cb with PID 1 (/application/BOOT-INF/classes started by root in /application)
2022-09-05 08:36:35.179  INFO 1 --- [           main] o.s.s.petclinic.PetClinicApplication     : No active profile set, falling back to 1 default profile: "default"
2022-09-05 08:36:36.144  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2022-09-05 08:36:36.187  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 34 ms. Found 2 JPA repository interfaces.
2022-09-05 08:36:36.823  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2022-09-05 08:36:36.832  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-09-05 08:36:36.833  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.64]
2022-09-05 08:36:36.927  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-09-05 08:36:36.927  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1661 ms
2022-09-05 08:36:37.355  INFO 1 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'vets' created in EhcacheManager.
2022-09-05 08:36:37.363  INFO 1 --- [           main] org.ehcache.jsr107.Eh107CacheManager     : Registering Ehcache MBean javax.cache:type=CacheStatistics,CacheManager=urn.X-ehcache.jsr107-default-config,Cache=vets
2022-09-05 08:36:37.391  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2022-09-05 08:36:37.602  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2022-09-05 08:36:37.765  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2022-09-05 08:36:37.807  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 5.6.9.Final
2022-09-05 08:36:37.950  INFO 1 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2022-09-05 08:36:38.051  INFO 1 --- [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2022-09-05 08:36:38.693  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2022-09-05 08:36:38.701  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2022-09-05 08:36:40.153  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 13 endpoint(s) beneath base path '/actuator'
2022-09-05 08:36:40.222  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2022-09-05 08:36:40.243  INFO 1 --- [           main] o.s.s.petclinic.PetClinicApplication     : Started PetClinicApplication in 5.446 seconds (JVM running for 5.782)

Our exploded Spring Boot app started in just under 6 seconds, which is quite impressive! Let's compare it to the original Dockerfile that used the far jar approach:

$ # With fat jar approach

$ docker run -p 8080:8080 petclinic


              |\      _,,,--,,_
             /,`.-'`'   ._  \-;;,_
  _______ __|,4-  ) )_   .;.(__`'-'__     ___ __    _ ___ _______
 |       | '---''(_/._)-'(_\_)   |   |   |   |  |  | |   |       |
 |    _  |    ___|_     _|       |   |   |   |   |_| |   |       | __ _ _
 |   |_| |   |___  |   | |       |   |   |   |       |   |       | \ \ \ \
 |    ___|    ___| |   | |      _|   |___|   |  _    |   |      _|  \ \ \ \
 |   |   |   |___  |   | |     |_|       |   | | |   |   |     |_    ) ) ) )
 |___|   |_______| |___| |_______|_______|___|_|  |__|___|_______|  / / / /
 ==================================================================/_/_/_/

:: Built with Spring Boot :: 2.7.1


2022-09-05 08:40:30.802  INFO 1 --- [           main] o.s.s.petclinic.PetClinicApplication     : Starting PetClinicApplication v2.7.0-SNAPSHOT using Java 17.0.4.1 on 93195813a338 with PID 1 (/app.jar started by root in /)
2022-09-05 08:40:30.805  INFO 1 --- [           main] o.s.s.petclinic.PetClinicApplication     : No active profile set, falling back to 1 default profile: "default"
2022-09-05 08:40:32.252  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2022-09-05 08:40:32.331  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 60 ms. Found 2 JPA repository interfaces.
2022-09-05 08:40:33.147  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2022-09-05 08:40:33.158  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-09-05 08:40:33.159  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.64]
2022-09-05 08:40:33.247  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-09-05 08:40:33.247  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2385 ms
2022-09-05 08:40:33.774  INFO 1 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'vets' created in EhcacheManager.
2022-09-05 08:40:33.789  INFO 1 --- [           main] org.ehcache.jsr107.Eh107CacheManager     : Registering Ehcache MBean javax.cache:type=CacheStatistics,CacheManager=urn.X-ehcache.jsr107-default-config,Cache=vets
2022-09-05 08:40:33.837  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2022-09-05 08:40:34.063  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2022-09-05 08:40:34.240  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2022-09-05 08:40:34.290  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 5.6.9.Final
2022-09-05 08:40:34.463  INFO 1 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2022-09-05 08:40:34.587  INFO 1 --- [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2022-09-05 08:40:35.302  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2022-09-05 08:40:35.311  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2022-09-05 08:40:37.049  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 13 endpoint(s) beneath base path '/actuator'
2022-09-05 08:40:37.117  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2022-09-05 08:40:37.138  INFO 1 --- [           main] o.s.s.petclinic.PetClinicApplication     : Started PetClinicApplication in 6.84 seconds (JVM running for 7.272)

Our application is slower by almost 1.5 seconds while running on the same machine and the same Docker engine. Add to that the overhead of pulling and storing an entire 150mb+ layer on every single change from a remote registry, and you'll quickly understand the benefits of this approach.

Customizing the layers

You might have your own set of release dependencies that are not part of the dependencies provided by Spring. For example, your company might be using an internal set of dependencies for security, logging and auditing that you need to include in every Spring Boot application. It might make sense to extract such dependencies into their own layer as well since they may change more often than the Spring dependencies. To do it, you can create a custom layers configuration:

<layers xmlns="http://www.springframework.org/schema/boot/layers"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.springframework.org/schema/boot/layers
                          https://www.springframework.org/schema/boot/layers/layers-2.7.xsd">
    <application>
        <into layer="spring-boot-loader">
            <include>org/springframework/boot/loader/**</include>
        </into>
        <into layer="application" />
    </application>
    <dependencies>
        <into layer="snapshot-dependencies">
            <include>*:*:*SNAPSHOT</include>
        </into>
        <into layer="custom-security-dependencies">
            <include>com.mycompany.security:*</include>
        </into>
        <into layer="dependencies"/>
    </dependencies>
    <layerOrder>
        <layer>dependencies</layer>
        <layer>spring-boot-loader</layer>
        <layer>snapshot-dependencies</layer>
        <layer>custom-security-dependencies</layer>
        <layer>application</layer>
    </layerOrder>
</layers>

To apply this configuration in your application, point the spring-boot-maven-plugin to the custom layer configuration:

<project>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <layers>
                        <enabled>true</enabled>
                        <configuration>${project.basedir}/src/layers.xml</configuration>
                    </layers>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Don't forget to change your Dockerfile accordingly:

FROM eclipse-temurin:17-jre-jammy as builder
WORKDIR application
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM eclipse-temurin:17-jre-jammy
WORKDIR application
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/custom-security-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

Avoid using uber jars in your Dockerfiles. Use Spring's builtin layertools to explode the jar for more efficient packaging in an OCI image.

Setting bootstrap arguments

In the example above, we ran the Java application with everything as default:

ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

These defaults may be ok, but they're probably not. There are too many variables that are taken into consideration by the JVM to decide on things like the heap size, available memory and garbage collection algorithm. Always profile your app before production and apply settings that you feel were validated in production-like environments. At the last section of this article I will include some profiling tips that may help you.

There are many ways to pass arguments to the JVM. In containerized environments, it's best to use the JAVA_TOOL_OPTIONS environment variable since it's the safest choice that can be read by the JVM inside a container. No, do not use JAVA_OPTS - this is a shell script convention and the JVM doesn't honor it.

Some of the more important arguments to pass to the JVM have to do with memory management. You'll need to take several things under consideration:

• Your JVM memory is (mostly) a combination of the heap memory and the non-heap memory (Stack Memory, Direct Memory etc.).
• Most of your application's memory will be allocated to the heap. How much memory should you assign to it? It depends!
• How much memory does the container has?
• How many classes are in-memory?
• How many threads are running at the same time (including Thread Pools)?

• How much memory is taken up by the Stack memory?

• Your application may also use a considerable amount of non-heap memory for things like NIO Direct Memory. For example, Kafka and many other socket-based frameworks make use of NIO for better performance. Without setting proper arguments, it might kill your JVM without you realizing what happened (as I learned the hard way...). Profile your application for direct memory by including the JVM arguments -XX:NativeMemoryTracking=summary-XX:+PrintNMTStatistics. Add the resulting value to the total memory footprint calculations.

At Pivotal (who later became VMware Tanzu), we developed a useful little tool that helped us with these memory calculations. Originally developed for Cloud Foundry, the Java Buildpack Memory Calculator was created to help developers on the platform, who really just wanted to cf push their Spring application and have the platform build the image for them without thinking too much on production optimizations. This tool still exists and provides a great starting point for you to consider when setting memory arguments.

The calculator takes as input the parameters from above like expected Stack memory, expected number of classes, expected number of concurrent threads etc. You still need to profile your app to get some understanding of what these values mean in your use case. The calculator would then output the recommended max heap size based on our research and real-world experience. The algorithm is clearly documented in the github repo.

For example, let's say my application has 750 classes, my container has 1G of total memory, I expect my stack size to be around 1Mb, and I calculated the total threads used by my application to be around 100 threads, give or take. I'll run the command as follows:

$ ./java-buildpack-memory-calculator --loaded-class-count=750 --thread-count=100 --total-memory=1024M --jvm-options "-Xss1024k"

-XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=17919K -XX:ReservedCodeCacheSize=240M -Xmx672256K

The memory calculator gave a recommended max heap size of around 670 megabytes, based on the algorithm total memory - (headroom + direct memory + metaspace + reserved code cache + (thread stack * thread count)). To clarify how important the thread pool is to this calculation, let's imagine our application uses Tomcat and utilizes 200 additional threads for Web requests, bumping the total number of threads to 300. The calculation is now:

$ ./java-buildpack-memory-calculator --loaded-class-count=750 --thread-count=300 --total-memory=1024M --jvm-options "-Xss1024k"

-XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=17919K -XX:ReservedCodeCacheSize=240M -Xmx467456K

Our max heap size is now down to around 470 megabytes.

Notice that the tool always gives a default value for MaxDirectMemorySize of 10M. That's because there's really no way of telling the true amount to be used without profiling the application, so it chose some sane default. Remember to check your direct memory usage as mentioned above. It could have a meaningful impact your heap size. Let's say our application uses Kafka with many concurrent consumers, and therefore requires 64 MB for direct memory. The calculation will now be:

$ ./java-buildpack-memory-calculator --loaded-class-count=750 --thread-count=300 --total-memory=1024M --jvm-options "-Xss1024k -XX:MaxDirectMemorySize=64M"

-XX:MaxMetaspaceSize=17919K -XX:ReservedCodeCacheSize=240M -Xmx412160K

Our heap size is now down to just ~410 megabytes. Perhaps 1 gigabyte for the container is not enough at this point?

As mentioned before, the JVM memory arguments are entirely dependent on the target deployment environment and available container memory. You wouldn't want to hardcode these values into the Dockerfile, since they keep changing. You have three options here:

1. Set the JAVA_TOOL_OPTIONS environment variable as part of your Kubernetes deployment yaml (more on this later).
2. Calculate the arguments when the container starts, but before running the actual Spring-based application. The next section shows how this is done.
3. Use a mixture of options 1 and 2. Set what you can in your deployment manifest, and let the container additional calculate arguments on startup.

Who needs Dockerfiles?

It's almost 2023. You don't compile your own Linux kernels anymore, I would hope. Why are you still writing Dockerfiles?

Docker images (well, OCI images, to be exact) are great. They are the golden standard for container packaging that can run anywhere. But Dockerfiles are really just scripts to build these images. It's been around since the time that OCI images became popular, so they are considered the default. We have other ways to get a working OCI image, and some of options are more focused on the what than on the the how. One of these tools is jib. It's developed at Google and can integrate with your build tools to automatically create a Java distroless container image for you without the hassle of writing the perfect Dockerfile. But there's also a standard that can be used for a wider context than just Java images.

Cloud Native Buildpacks

Cloud Native Buildpacks (CNBs) is a specification for how to build OCI images without Dockerfiles, and I would argue it has three major advantages going for it:

• It's a CNCF project.
• It will proactively patch your base Linux distribution and your OpenJDK (or other middleware runtimes for other languages) on each new build.
• It's directly integrated into the existing Spring build plugins!

TL;DR, the following will generate a production-ready OCI image for you:

mvn spring-boot:build-image

This command will use many of the things I mentioned above:

• It will create an optimized OCI image based on the layers listed by the layertools command (unless you're on Spring Boot 2.3, where it wasn't the default).
• It will run the Java Buildpack Memory Calculator to calculate the correct heap size.
• On container startup, it will set production-ready defaults for JAVA_TOOL_OPTIONS, taking into account user-provided environment variables and the memory calculation. Here's the boot logs of a container built by mvn spring-boot:build-image, running on a Mac with Docker Desktop that has resource configuration of 4 CPUs and 4 GB assigned memory. Notice that JAVA_TOOL_OPTIONS is calculated and set before Spring even boots, based on the available container memory:

> docker run -p 8080:8080 spring-petclinic:2.7.1-SNAPSHOT
Setting Active Processor Count to 4
Calculating JVM memory based on 3198728K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx2572558K -XX:MaxMetaspaceSize=114169K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 3198728K, Thread Count: 250, Loaded Class Count: 17743, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 127 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx2572558K -XX:MaxMetaspaceSize=114169K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true


              |\      _,,,--,,_
             /,`.-'`'   ._  \-;;,_
  _______ __|,4-  ) )_   .;.(__`'-'__     ___ __    _ ___ _______
 |       | '---''(_/._)-'(_\_)   |   |   |   |  |  | |   |       |
 |    _  |    ___|_     _|       |   |   |   |   |_| |   |       | __ _ _
 |   |_| |   |___  |   | |       |   |   |   |       |   |       | \ \ \ \
 |    ___|    ___| |   | |      _|   |___|   |  _    |   |      _|  \ \ \ \
 |   |   |   |___  |   | |     |_|       |   | | |   |   |     |_    ) ) ) )
 |___|   |_______| |___| |_______|_______|___|_|  |__|___|_______|  / / / /
 ==================================================================/_/_/_/

:: Built with Spring Boot :: 2.7.1
...
...

Running the same image in a Linux machine with less resources results in different memory values, as it should:

# docker run -p 8080:8080 harbor.apps.cf.tanzutime.com/apps/spring-petclinic:2.7.1-SNAPSHOT
Setting Active Processor Count to 1
JVM DNS caching disabled in favor of link-local DNS caching
Calculating JVM memory based on 1221856K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx595686K -XX:MaxMetaspaceSize=114169K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1221856K, Thread Count: 250, Loaded Class Count: 17743, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 127 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=1 -XX:MaxDirectMemorySize=10M -Xmx595686K -XX:MaxMetaspaceSize=114169K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true


              |\      _,,,--,,_
             /,`.-'`'   ._  \-;;,_
  _______ __|,4-  ) )_   .;.(__`'-'__     ___ __    _ ___ _______
 |       | '---''(_/._)-'(_\_)   |   |   |   |  |  | |   |       |
 |    _  |    ___|_     _|       |   |   |   |   |_| |   |       | __ _ _
 |   |_| |   |___  |   | |       |   |   |   |       |   |       | \ \ \ \
 |    ___|    ___| |   | |      _|   |___|   |  _    |   |      _|  \ \ \ \
 |   |   |   |___  |   | |     |_|       |   | | |   |   |     |_    ) ) ) )
 |___|   |_______| |___| |_______|_______|___|_|  |__|___|_______|  / / / /
 ==================================================================/_/_/_/

:: Built with Spring Boot :: 2.7.1
...
...

Security Patching

Cloud Native Buildpacks also have the added value of proactive security patching, which is an incredible feature to have in production. Every time you execute the plugin, Spring downloads the latest Paketo base builder image:

[INFO] --- spring-boot-maven-plugin:2.6.3:repackage (repackage) @ spring-petclinic ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] <<< spring-boot-maven-plugin:2.6.3:build-image (default-cli) < package @ spring-petclinic <<<
[INFO]
[INFO]
[INFO] --- spring-boot-maven-plugin:2.6.3:build-image (default-cli) @ spring-petclinic ---
[INFO] Building image 'docker.io/library/spring-petclinic:2.6.0-SNAPSHOT'
[INFO]
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder:base' 0%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder:base' 5%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder:base' 7%
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder:base' 10%
...
[INFO]  > Pulling builder image 'docker.io/paketobuildpacks/builder:base' 100%

The builder image is constantly updated with the latest security patches. Image building is done entirely inside this builder container. Every time a new buildpack is available with the latest security patches, it will be applied to the newest build.

[INFO]  > Running creator
[INFO]     [creator]     6 of 24 buildpacks participating
[INFO]     [creator]     paketo-buildpacks/ca-certificates   3.4.0
[INFO]     [creator]     paketo-buildpacks/bellsoft-liberica 9.8.0
[INFO]     [creator]     paketo-buildpacks/syft              1.20.0
[INFO]     [creator]     paketo-buildpacks/executable-jar    6.5.0
[INFO]     [creator]     paketo-buildpacks/dist-zip          5.4.0
[INFO]     [creator]     paketo-buildpacks/spring-boot       5.19.0

The resulting image is built from the latest Run base image, which is also constantly patched:

[INFO]     [creator]     Saving harbor.mysite.com/apps/spring-petclinic:2.6.0-SNAPSHOT...
[INFO]     [creator]     *** Images (179aab23f421):
[INFO]     [creator]           harbor.mysite.com/apps/spring-petclinic:2.6.0-SNAPSHOT

This process continues on every new rebuild. Pull the latest patched Builder, apply the latest patched buildpacks, use the latest patched Run image.

If you want to do this process at scale, you can run these builds directly on Kubernetes using the kpack open source project. With kpack, you will have the ability to retroactively patch the base OS without running full rebuilds, using a process called rebasing. kpack has an Enterprise-ready version called Tanzu Build Service that is more suited for enterprises and locations that have no internet access.

Cloud Native Buildpacks in different Spring Boot versions.

There are some differences that you should be aware of regarding the build-image implementation:

1. If you're still using Spring Boot 2.3 (why?), the resulting image will not use the layered approach as described above. In Spring Boot 2.4 and later, this is the default. In order to gain the benefits of better OCI layering in 2.3, you'll need to provide the following configuration in your pom.xml:

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <layers>
                        <enabled>true</enabled>
                    </layers>
                </configuration>
            </plugin>
   

2. In Spring Boot 2.4 and above, there's an additional enhancement you get for free: The build-image process will also remove all the spring-boot-starter jars from the target image. These are really only needed at compilation time and not at runtime, so removing them results in smaller images and in better performance.
3. In Spring Boot 2.4 and above, you can configure your own custom registry for the target resulting image. Spring Boot will push the image to the remote registry for you as part of the build process - great for CI pipelines. For example:

   <project>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <name>harbor.mysite.com/apps/spring-petclinic</name>
                        <publish>true</publish>
                    </image>
                    <docker>
                        <publishRegistry>
                            <username>my-username</username>
                            <password>my-password</password>
                        </publishRegistry>
                    </docker>
                </configuration>
            </plugin>
        </plugins>
    </build>
   </project>
   

4. Starting in Spring Boot 2.5 and later - you can also customize the buildpack that is used to create the resulting image. The Spring Boot plugin uses the Paketo Java buildpack by default. With this new feature, you can use buildpacks from other vendors such as those provided by Google or VMware Tanzu, or even write your own. To reference a different set of buildpacks, use the following configuration:

   <project>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <buildpacks>
                            <buildpack>file:///path/to/example-buildpack.tgz</buildpack>
                            <buildpack>urn:cnb:builder:paketo-buildpacks/java</buildpack>
                        </buildpacks>
                    </image>
                </configuration>
            </plugin>
        </plugins>
    </build>
   </project>
   

5. In Spring Boot 2.5 and later, you can create additional image tags for the resulting image, to support better image management that is more suited to your organization. For more information, check out the plugin documentation parameters.

Use Cloud-Native Buildpacks if you can. You will get security patches, simplified workflow, and production-ready defaults.

Running the image

Finally, we have a working OCI image that we can use to run on Kubernetes.

But how do we run this thing?

There are many factors that need to go into consideration when running our image in Kubernetes. Let's review them one by one.

12 Factor apps

The concept of developing 12 factor apps is still as relevant as ever. Even if some factors seem obvious when running in containers and on Kubernetes, it's still good to have this north star. Things like dev/prod parity, scaling out via the process model (more containers!), disposability, outputting logs to the standard output, all fit perfectly into the Kubernetes model. There are those who advocate that in this day and age, we need a few more of these factors. Although all of these factors are important, when it comes to Kubernetes there is one thing to consider above all else, so much so that it is sometimes referred to as the "1-Factor app":

Can it be restarted gracefully?

If it can, then you can let Kubernetes do a lot of things behind the scenes even if your application is not written perfectly according to all the best practices.

Liveness and readiness probes

To make sure your Spring application runs well on Kubernetes, you need understand how Kubernetes interacts with your application during various lifecycle phases. This starts with having your Spring application notify the Kubernetes runtime that it is running and ready to accept traffic.

Actuator health endpoint

Spring Boot has has an endpoint that is designed for something like this. Adding spring-boot-starter-actuator to your classpath will allow you to have various production-ready endpoints that report health and metrics about of your application. By default, the health endpoint is always exposed. In earlier versions of Spring Boot the info endpoint was also exposed by default, but this has since been removed. With the actuator starter in your classpath, you can now add the following to your Kubernetes deployment manifest in order to let Kubernetes know when your application is ready to accept traffic and if it is still considered "alive":

    spec:
      containers:
          ...
          livenessProbe:
            httpGet:
              port: 8080
              path: /actuator/health
          readinessProbe:
            httpGet:
              port: 8080
              path: /actuator/health

This is a good starting point. We're telling our application to accept traffic only when the Spring Boot actuator health endpoint returns 200 OK. If during runtime the endpoint returns an error, we also tell Kubernetes to restart our pod. But there are caveats... as always.

1. Do you really want to restart your pod whenever the health endpoint returns an error? The default actuator endpoint usually returns the overall health of your application, including whether connectivity to a database or a message broker is working as expected. If you cannot access your database, restarting your application won't do much good beyond adding more strain on your (probably overloaded) database when new connection pools are created on each restart, which might actually cause a cascading failure. Perhaps implementing circuit breakers in your application and add alerting in your monitoring system is a better approach. It does make more sense to set readiness probes based on external components to the application, but even then, "it depends". You might be ok with some Redis cache being inaccessible on startup, but not ok with the main database being inaccessible.

2. It's considered a best practice to assign a separate port to the actuator endpoints. That is because these endpoints can reveal sensitive metrics and information about your application, so moving these to a different port allows you to easily block access to this information, while keeping the main application port open to an external service or ingress. This is especially true for distributed and multi-tenant environments such as Kubernetes. To set a different management port, set the following property:

   management.server.port=9090
   

   However, there's a catch here - when it comes to readiness and liveness - you want these probes to exist on the main server application port, and not the management port. That is because probing the management port might result in false positives - the management port might be responding to your requests just fine, while the actual application is still trying to start up correctly.

3. Actuator health endpoints are mostly based around HTTP readiness. If your application is a batch application or a streaming application, you need to provide other verifiers to report if it is ready or not (here I come, you can't hide...).

Luckily, Spring has you covered.

Spring Liveness and Readiness Health Groups

Spring Boot 2.2 introduced the concept of Health Groups, allowing you to group multiple actuator health indicators under a single group. You can then define specific properties for that particular group only. Spring Boot 2.3 took this concept further by creating two health groups out of the box: a readiness group and a liveness group. To have these two groups enabled, the following setting must be enabled:

management.endpoint.health.probes.enabled=true

To make your life easier, Spring sets this property to true automatically when it detects the application runs in Kubernetes.

With this new setup, you get the full-blown and detailed health information including dependent resources under /actuator/health, and two separate sub-groups: /actuator/health/readiness and /actuator/health/liveness. Based on the reasons explained in item #1 above, Spring Boot by default does not rely on any external state for readiness and liveness, it is only checking the internal application state. Since these are now two separate groups, you can also customize each group with its own set of checks that is suitable for your application. For example, if your application must have database connectivity before it can report it is ready to accept traffic, you can add that check to the list:

management.endpoint.health.group.readiness.include=readinessState,db

You might even want to write your own custom checks if you have some unique readiness requirements that are not part of the built-in verifiers. For example, imagine that your application relies heavily on a third party SaaS API such as Twilio to be available before the application is ready to accept traffic. Write a custom verifier that checks the API endpoint and add it to the list:

management.endpoint.health.group.readiness.include=readinessState,externalTwilioVerifier

On the other extreme, if your application is a batch or a streaming application that doesn't accept web traffic at all, you might want to rely entirely on a specific external service such as a message broker and nothing else:

management.endpoint.health.group.readiness.include=rabbitHealthIndicator

So, we now have better control of what exactly does it mean to have our application "ready" and "alive".

But we are still conflicted with the fact that most actuator endpoints should live on a separate "management port" for security reasons, while we want our readiness and liveness probes to live on the application port. The Spring documentation for version 2.3 has a big yellow warning specifically on this:

If your Actuator endpoints are deployed on a separate management context, be aware that endpoints are then not using the same web infrastructure (port, connection pools, framework components) as the main application. In this case, a probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections).

Luckily, starting with Spring Boot 2.6, there is a good solution for this. You can now set a health group as an additional path to either the main application port or to the management port. For example:

management.endpoint.health.group.readiness.additional-path="server:/readyz"
management.endpoint.health.group.liveness.additional-path="server:/livez"

In addition to whatever was configured as the management port and path, this setting will also make the readiness endpoint available on the main (server:) port, under the HTTP path readyz, and make the liveness endpoint available on the main port under the HTTP path livez. To set a health group to the management port instead of the main server port, use management: as the prefix.

Some more liveness and readiness settings

• A container is considered alive or ready if it returns any HTTP response code between 200-399. Kubernetes doesn't really care what is returned as long as it is not 4xx and 5xx.

• Some additional settings that could be relevant when defining your liveness and readiness probes:

    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /livez
        port: 8080
      periodSeconds: 10
      timeoutSeconds: 1
      initialDelaySeconds: 5
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 1

• periodSeconds tells Kubernetes how often to poll the endpoint.
• timeoutSeconds tells Kubernetes when to timeout the HTTP request (in seconds) and consider it an error that requires the pod to be restarted.
• initialDelaySeconds indicates how long to wait before probing. This is especially important for Spring Boot applications that might require a long startup time. Profile your application to figure out the preferred values here. Beware of using an unnecessary long value, because your pod will not be available until that time has passed. Also note that the initialDelaySeconds for livenessProbe only begins polling after the application is marked as Ready.
• failureThreshold defines how many times a failed response is considered "ok". You probably don't want to immediately kill your pod if a temporary network glitch returned a bad gateway once or twice.

Use Spring's built-in readiness and liveness health probes. Keep the probes on the main application port, and set a separate management port for other actuator endpoints.

Graceful Shutdown

Ok, we took care of the startup lifecycle. But what about shutdown? In Kubernetes, shutdowns are inevitable. Whenever a cluster node is being upgraded, removed or resized, two things happen:

• A process called cordon prevents Kubernetes from scheduling new pods on the node (it's status is changed to SchedulingDisabled).
• A process called drain evicts all the pods from the node.
• Maintenance on the node can now be performed (delete/upgrade/resize etc.).

Another common use case for shutting down pods is when autoscaling is being utilized, such as when using a Horizontal Pod Autoscaler, or a runtime like Knative that even scales to zero when there is no traffic.

Regardless of the reason for the shutdown, we want to make sure that our application doesn't get killed abruptly when it is still processing in-flight HTTP requests or while executing queries against our database. If it gets killed too soon, some clients will receive errors and some database transactions might not be updated (or even corrupted, depending on the database).

Configuring graceful shutdown

By default, even in the latest versions of Spring, the Web server associated with the application will be shutdown immediately. Setting a graceful shutdown in Spring used to be quite cumbersome and required some custom solutions. Thankfully, starting with Spring Boot 2.3, graceful shutdown is supported out of the box:

server.shutdown=graceful

With this property set, when the container receives a SIGTERM to begin shutdown, Spring will block the associated Web server (Tomcat, Netty etc.) from accepting any new connections before actually shutting down. We now have a time window to handle in-flight requests without receiving new requests.

We now want to wait for all the in-flight requests to complete before exiting the Spring application. For this, we have a timeout property for the shutdown phase, which defaults to 30 seconds. To override this default, change the property as follows:

spring.lifecycle.timeout-per-shutdown-phase=20s

It is likely that you only want to set these parameters while running in the context of Kubernetes. Luckily, this is easily doable since Spring automatically adds a profile called kubernetes when it detects the application runs in Kubernetes:

spring.application.name: "SpringPetclinic"
---
spring.application.name: "SpringPetclinicKubernetes"
spring.config.activate.on-cloud-platform: "kubernetes"
spring.lifecycle.timeout-per-shutdown-phase=20s
server.shutdown=graceful

In the example above, the shutdown properties are only applied if the application runs in Kubernetes. Outside of Kubernetes, the server will shutdown immediately. Also, just for fun, we're changing the application name.

Prevent new requests coming to a pod that shuts down

Kubernetes is a distributed system working in event loops. As such, eventual consistency considerations must be taken into account. One classic example of this is the fact that a pod might begin the shutdown lifecycle, while other Kubernetes resources continue to route traffic to the pod. In order to block new requests from coming in to the pod when the shutdown flow begins, we can configure a preStop phase in our pod lifecycle:

spec:
  containers:
    image: odedia/spring-petclinic
    ...
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]
  terminationGracePeriodSeconds: 30

In this simple use case, we're telling our application to sleep for 10 seconds once the shutdown lifecycle begins. We allow other components this time window to figure out that our pod is no longer in the game. During that time, initially new requests might still come through to the main application running in the container, but eventually new requests should not be routed. You might need to tweak this wait time based on the size of your cluster and configuration. After this preStop phase completes, we are now at a stage where new requests are (hopefully) not being routed to our pod, while existing in-flight requests are being executed within the timeout-per-shutdown-phase window.

Termination grace period

The flow of a shutdown is as follows:

1. preStop is being executed (in our example we're basically waiting for 10 seconds for Kubernetes to get its resources in order with our shutdown).
2. Kubernetes sends a SIGTERM to our pod.
3. Our pod receives the signal and begins the shutdown process. For a graceful shutdown this means blocking any new requests from our Web server and waiting for in-flight requests to complete, up to the value of timeout-per-shutdown-phase.

Here's the caveat: the value for terminationGracePeriodSeconds defines how long Kubernetes will wait before violently killing your pod with a SIGKILL signal. The default is 30 seconds. But when should we start counting these 30 seconds? The answer is: from the beginning of the preStop phase. In practice, this means that you should set the terminationGracePeriodSeconds to a value higher than the sum of the preStop execution time + timeout-per-shutdown-phase time, otherwise your pod will always get killed abruptly. It also means that if you're using nothing but the defaults, you're in for a world of hurt, because the default timeout-per-shutdown-phase is 30 seconds and the default terminationGracePeriodSeconds is also 30 seconds! Great, another distributed systems race of "who dies first?".

Use Spring's graceful shutdown setting while running in Kubernetes. Make sure timeout settings are set correctly so they do not create conflicts.

CPU Requests and Limits

Now that our application adheres to the "1-Factor app" - that is, it can be restarted gracefully, we turn our head to the important discussion of CPU requests and CPU limits. As you recall, the JVM makes a lot of assumptions based on the available CPU and memory - things like which garbage collection to use, the size of Fork/Join thread pools, and various libraries will make decisions based on the value of Runtime.getRuntime().availableProcessors().

CPU requests/limits are measured in CPU millicores. 1000 millicores translates into 1 vCPU in a virtualized environment such as the public cloud or vSphere, or a physical hyperthread if you run on bare metal. This is another plus to running Kubernetes in virtualized environments, even if you have your own hardware in a data center, because you get an additional layer of abstraction that allows you to share and over-provision resources on the same hardware.

A CPU request in Kubernetes is the minimum guaranteed amount of CPU that Kubernetes will assign to your workload.

A CPU limit measures the maximum amount of CPU that your workload can utilize before being throttled.

It's important to note that CPU requests are not directly related to speed or performance. It may indirectly impact performance, but at its core, CPU requests are designed to help Kubernetes make better decisions on resource allocation. Nothing more.

Since CPU limits could eventually throttle your application, it could have performance implications.

Let's review the tradeoffs when setting various CPU requests and limits combinations:

1. CPU requests and limits are set to the same value: You have predictable and guaranteed CPU allocation for your pod. You might think this is exactly what you need, but if you think about how a typical Spring Boot / Java application behaves - it usually does a lot more on startup in order to create thread pools, scan for Spring beans and other startup activities. This warm-up time will greatly benefit from more CPU cores than are needed during normal operation.

2. CPU requests and CPU limits are not set: Welcome to the wild west. You have zero predictability. Your pods can be killed at any point when new workloads that do have CPU requests need to allocate available capacity. Specifically for JVM workloads before JDK 19, it will actually mean that you limit each container to just a single vCPU. For workloads on JDK 19 and later, it will try to use as many vCPUs as available on the host. Please don't use this setting.

3. CPU requests are set to a value lower than the CPU limits: You get guarantees about the typical CPU consumption that your application needs, but can also burst up to the defined CPU limits during warmup time or if a sudden spike in traffic reaches your pod.

Option 3 does seem like the best of both worlds. You have guaranteed resources that will not be taken away when other resources request CPU capacity on the worker node. In fact, requests (which are based on container CPU shares) are relative. This means that they are only taken into consideration when there are constraints on the worker node from other competing resources. Otherwise, the container can use as much CPU as it needs up to the host limit, or the configured value for CPU limits. So, this begs the question: what should you set as the CPU limits?

How about No CPU Limits?

As this blog post wonderfully articulates, you really don't get any meaningful benefits by assigning CPU limits. You're only preventing your containers from using the excess CPU capacity that was not assigned to other containers. Why would you ever want to artificially prevent your workloads from utilizing unused CPU cycles? Perhaps it makes sense to set limits in a staging/performance environments where you want have predictable results to make decisions about production, but once you're in production you really want to squeeze every bit of available CPU cycles as you could possibly get. Why limit yourself?

Besides, the whole mechanism that Kubernetes used to assign CPU quotas/limits is based on the amount of time you're getting before being throttled, relative to the total number of cores on the worker node. Do you have nodes in production with different CPU capacities, or have some worker nodes with more containers compared to others? You'll get different results on each of these worker nodes. In short - the results will be unpredictable.

Still not convinced? This set of blog posts [1] [2] [3] goes deep into the underpinnings of Kubernetes resources, and also reaches the same conclusion:

"Most of us are trying to achieve the best possible performance on the cheapest infrastructure while minimizing downtime. Production workloads should be able to utilize idle CPU. Containers won’t “steal” CPU from other containers if you set your CPU requests right, if you didn’t set CPU requests or set them badly, I’m afraid that the CPU limit wouldn’t save you."

If that was not enough, here's a recommendation from Tim Hockin, one of the original founders of the Kubernetes project at Google:

The important setting is the CPU requests - this will tell Kubernetes how many resources to allocate for your workload at a minimum. Setting this to too low could result in bad performance if you have a lot of other workloads on the node, and will also get you closer to the wild west scenario, since all workloads will fight for maximum resources that they should have allocated in the first place. Setting this too high could result in wasted resources and having fewer workloads running on the each node. Profile your apps before production!

Here's a snippet of CPU configuration from the Spring Petclinic deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-petclinic
...
spec:
  template:
    spec:
      containers:
      - image: harbor.mysite.com/app/spring-petclinic:v1
        name: spring-petclinic
        resources:
          requests:
            cpu: "2000m"

Important note: if you're using a fully managed Kubernetes solution such as Google Autopilot, you don't have this level of control for your deployments. Google Autopilot specifically will set the CPU limits to the same value of the CPU requests, regardless of what you configured. This might change in the future, but the bottom line is that you don't have as much control with these managed solutions, since they prefer predictability above all else. If you want to have complete control, use an upstream vanilla Kubernetes distribution such EKS, GKE, AKS or Tanzu Kubernetes Grid.

Things get complicated

If you think everything is good, I have some bad news - things keep changing between JDK versions and could have untrustworthy behaviour based on a combination of your JDK, your CPU requests/limits settings, and available resources.

Here's one example: a very recent bug report in JDK 19 and later JDK-8281571 suggests not to use CPU shares to compute the JVM active processor count, and instead use the CPU limits (whaaat?!).

To understand why that is, we need to understand how Kubernetes uses assigned CPU requests, which are translated to CPU shares per container.

CPU requests are the guaranteed value. But, as briefly mentioned previously, if nothing else runs on the node, you'll get more resources. That's why we didn't want to set a limit in the first place. Let's review this with a sample 16-core worker node:

• For a single container on the node requesting 4000 millicores, the container will actually get all 16 cores because nothing else runs on the node. So, each requested core is translated to 16/4 actual cores --> 4 actual cores for each requested core.
• When a second container requesting 4000 millicores joins the node (total of 8000 millicores requested on the node), each container will now receive 8 cores. Each requested core is now translated to 16/8 actual cores --> 2 actual cores for each requested core.
• When a third container requesting 2000 millicores joins the node (total 10 cores requested on the node), the first and second containers will receive 6400 millicores each, and the third container will get 3200 millicores. Each requested core is now translated to 16/10 actual cores --> 1.6 actual cores for each requested core.

For the JDK, the CPU shares suddenly become an unreliable value. The computed value for ActiveProcessorCount will drift over time.

Houston, we have a problem.

Actually, we have two! The bug report also provides an example that shows a very problematic setting of CPU shares: 1024, which is the default setting if no CPUs were requested. While that should be translated to just 1 vCPU, it is actually being translated to (again! 🤦🏻‍♂️) the total number of available cores on the host machine:

 - docker run ... --cpu-shares=512 java ....    ==> os::active_processor_count()  = 1
 - docker run ... --cpu-shares=1024 java ...   ==> os::active_processor_count()  = 32 (total CPUs on this system)
 - docker run ... --cpu-shares=2048 java ...   ==> os::active_processor_count()  = 2

This all seems like one big mess, isn't it?

Thankfully, there's a relatively simple solution to this: Don't let the JVM decide the number of active processors automatically. Profile the actual usage needed for your app, and just set it yourself as part of the JAVA_TOOL_OPTIONS parameter -XX:ActiveProcessorCount. You have the power, and the JVM will honor the setting regardless of how many actual vCPUs are given to it by the container. You can set the CPU requests to 2000 millicores, and the ActiveProcessorCount to 4 for bursting. It will work.

Set CPU requests after profiling your application. Do not set CPU limits at all. Always set the active processor count explicitly.

Memory Requests and Limits

So we have pretty good idea of what to do with CPU requests and limits. Should the same be applied to memory requests and limits?

Probably not. CPU constraints are a considered a compressed resource in Kubernetes. This means that if needed, Kubernetes can throttle your pod to make "room" for other workloads. The same cannot be said for memory. If you go past your memory limit, the container will be terminated.

Let's review the options we have for setting memory requests and limits:

1. Memory requests and limits are set to the same value: You are guaranteed the requested amount of RAM even if the worker node is low on memory.
2. Memory requests is less than memory limits: If you are using more than the memory requested but less than the memory limit, and the worker node requires memory for other workloads, your pod will be terminated. Once memory is given to a pod, it cannot be taken away without restarting that pod.
3. Memory request and memory limits are not set: Again, this is the wild west scenario. Pods will be terminated when a worker node is low on memory.

This one seems simpler:

For Java workloads, always set the memory requests and memory limits to the same exact value.

Here's a snippet for both the CPU and memory configuration from the Spring Petclinic deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-petclinic
...
spec:
  template:
    spec:
      containers:
      - image: harbor.mysite.com/app/spring-petclinic:v1
        name: spring-petclinic
        resources:
          requests:
            memory: "2000Mi"
            cpu: "2000m"
          limits:
            memory: "2000Mi"

We now have a good setup for the container's memory. But what about the JVM's memory configuration? By default, the JVM sets the heap size to 1/4 of the available memory, which might not be the best default. What should we pass to the JAVA_TOOL_OPTIONS environment variable? Remember that your JVM lives in a container now, so your JVM memory should be a portion of the total available memory given to the container itself. The answer, as always, is that it depends. You could probably get away with setting the JVM memory to be a percentage of the container's memory. For example, the argument -XX:MaxRAMPercentage=75.0 will set the JVM's Max heap memory to 75% of the container's memory, but this seems too generalized. There are different memory regions created by the JVM (Heap, Stack, DirectMemory etc.). Use something like the Java Memory Calculator mentioned in the previous section to calculate the correct values. Apply the results as an environment variable in the deployment manifest, or (if you use cloud-native buildpacks), set some values in the deployment manifest and let cloud-native buildpacks calculate the rest on startup.

Putting it all together

We now have a pretty solid understanding of what JVM arguments to pass to JAVA_TOOL_OPTIONS related to memory. Add to that the parameter for -XX:ActiveProcessorCount, and we're in a good place.

You may also consider explicitly setting which garbage collector to use. If you set the active processor count to a single core, it will default to SerialGC. If you have 2 cores and approximately 2 GB of RAM, you'll get ParallelGC or G1GC based on the JVM version you're using. To avoid ambiguity, set it explicitly using -XX:+UseG1GC or -XX:-UseParallelGC, and other related arguments. If you don't have multiple cores available for parallel garbage collection, stick to SerialGC with -XX:-UseSerialGC.

When profiling your application, consider scaling up (CPU and RAM) before scaling out. You may be surprised by the performance and even cost benefits fewer beefy containers provide compared to many small containers.

My colleague Adib Saikali has a very good summary for all of this:

Spring Cloud

Spring Cloud is a popular set of components that is designed to ease the development of microservices based on Spring Boot. It began with a set of components that originally came from Open Source Netflix projects such as Eureka for service discovery and Hystrix for circuit breaking (later replaced by the Spring Cloud Circuit Breaker abstraction), and evolved over the years to include projects such as Spring Cloud Stream for event-driven applications, Spring Cloud Sleuth for distributed tracing, and Spring Cloud OpenFeign for simplified HTTP clients. While most of these Spring Cloud projects still make sense in a Kubernetes world, two of them stand out: Spring Cloud Eureka and Spring Cloud Config Server. These two solutions were developed before Kubernetes even existed, and were designed to solve two important pain points for microservices:

• How do I find other microservices in a distributed system, without having to manually manage a list of IP addresses or DNS records?
• How do I externalize my configuration to a central repository so my application can remain portable between environments?

Along with Spring Cloud Eureka, Spring Cloud Ribbon (and later Spring Cloud LoadBalancer) provided the ability call the available instances of a given microservice, based on load balancing rules such as simple round robin or advanced weighted-based routing.

Here's a typical Spring Cloud architecture without Kubernetes:

However, in Kubernetes there is already built-in solutions for service discovery and configuration management.

The built-in Kubernetes Service resource allows you to expose a set of pods/deployments behind a single internal cluster-wide IP address. It also assigns an internal DNS record to the service so it can be easily found inside the same namespace and even across namespaces in the same cluster.

For external configuration, Kubernetes provides the concept of ConfigMaps to manage a set of properties, and the notion of a Kubernetes Secret to configure sensitive data such as usernames and passwords.

If you can, prefer the Kubernetes-native solution. It will work interchangeably with other resources in the cluster written in other languages, and simplify your code since you don't have to use Spring Cloud dependencies at all.

• For service discovery - things are rather simple: for a given Kubernetes Service my-other-service, simply invoke the URL http://my-other-service and Kubernetes will take care of routing your Spring client application to the target server, load balancing requests based on the available number of instances. By default, you will lose the ability to perform sophisticated client-side load balancing operations such as weighted routing (unless you use Service Mesh solutions like Istio), but for the vast majority of use cases, it is more than adequate. If your service lives in another namespace, and provided you have the appropriate permissions assigned to the service account that runs your Spring application, you can make HTTP calls using http://my-other-service.the-other-namespace.svc.cluster.local.

• For external configuration, you can fairly easily create a ConfigMap from your existing application.properties or application.yaml and mount it to your deployment:

$ kubectl create configmap my-spring-config --from-file=./k8s/application.properties

The result will be your application.properties or application.yaml configured as a single literal in the resulting ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-spring-config
data:
  application.properties: |
    database=h2
    management.endpoints.web.exposure.include=*
    logging.level.org.springframework=INFO

The deployment can then be mounted to your Spring deployment manifest as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: spring-petclinic
  name: spring-petclinic
spec:
  replicas: 1
  template:
    spec:
      containers:
        - image: harbor.mysite.com/app/spring-petclinic:v1
          name: spring-petclinic
          ...
          env:
            - name: SOME_ENV_VARIABLE
              value: hello
          volumeMounts:
            - name: config-volume
              mountPath: /workspace/config
      volumes:
        - name: config-volume
          configMap:
            name: my-spring-config

In the deployment manifest above, we mounted the ConfigMap containing application.properties and also set some environment variable directly in the deployment yaml. Just because we can.

Here's the resulting Spring Boot application with Kubernetes-native services:

One could argue that Spring Cloud Config is better for auditing because your configuration is usually saved to a git repo which has version control and commit history, but that should also be true for your ConfigMaps and other Kubernetes resources (except secrets which should be saved in other secured storage such as Vault).

The algorithm that Spring uses to load properties from git does have more advanced logic than a simple ConfigMap. It allows for interesting topologies where generic configuration resides in a default application.properties while a microservice-specific configuration resides at a my-microservice.yaml or even my-microservice-forProfile.yaml. Still, these things are also (mostly) possible with a mix of ConfigMaps, Secrets and deployment environment variables.

You do lose one thing when you opt for Kubernetes-native Configuration: if your ConfigMap changes during runtime, the volume changes immediately but your Spring application will not get the updated values until it is restarted. With Spring Cloud Config, you had the ability to invoke POST /actuator/refresh on each of your instances, or to register for configuration change notifications using Spring Cloud Bus backed by a message broker. Both solutions would update your configuration at runtime without requiring a restart.

With Kubernetes ConfigMaps, all you can really do is restart your instances. If you have multiple instances of the application for high availability (as you should in production), you shouldn't experience downtime. You can run a rolling restart command when a ConfigMap has changed as follows:

kubectl rollout restart deployment/spring-petclinic

A rolling restart adheres to all the lifecycle phases we described above including the preStop hook and gracefulShutdown, so it could take some time to complete, but at least your application will not experience any downtime.

This all seem great for greenfield applications, but the reality is that there are a lot of applications out there that are already invested in Spring Cloud Eureka and in Spring Cloud Config. Migrating all of these brownfield applications to their Kubernetes-native counterparts could be quite the challenge, especially if your code relies heavily on annotations related to Eureka or if your git repo containing the configuration has a complex loading hierarchy.

Luckily, the Spring Cloud team has a solution for such applications, in the form of Spring Cloud Kubernetes. You can remove all references to spring-cloud-starter-config and spring-cloud-starter-netflix-eureka-server from your classpath, and replace them with the spring-cloud-kubernetes-all dependency. This dependency is a drop-in replacement that will let you easily migrate your Spring Cloud application to a Kubernetes world:

• The discovery client will now query the Kubernetes API server for available resources, instead of querying a Eureka server.
• Spring Cloud Config Client will look for attached ConfigMaps for loading the configuration, instead of relying on a Config Server.

These features are only enabled if you have the kubernetes profile configured. Luckily, when deploying a Spring boot application to Kubernetes, Spring adds this profile for you automatically.

However, there are things you should be aware of when deploying this sort of solution to production:

By default, all the instances that utilize the Spring Cloud Config Client or Spring Cloud Discovery Client will need access to the Kubernetes control plane's API server. This means that the service account used to run the Spring pod will need permissions to access the Kubernetes control plane.

You will also need to setup a Kubernetes NetworkPolicy that allows access to the Kubernetes API Server. My colleague Stuart Charlton explains it as follows:

1. The Spring Boot with the Spring Cloud Kubernetes dependency calls the Kubernetes API Server to read the ConfigMap.
2. The API endpoint Spring Cloud Kubernetes uses for this is the internal service DNS for the control plane. That is to say - kubernetes.default.svc.cluster.local. This is an internal, cluster-bound service with a custom endpoint, pointing at the Kubernetes API IP and port.
3. The flow is: Pod network IP --> K8s Internal Service (kubernetes.default.svc.cluster.local) --> The K8s API Server.

So, your Kubernetes Egress CIDR block will need access to both the external API server IP and port, and potentially (depending on the Container Network Interface/CNI you have in your cluster), to the internal Kubernetes service kubernetes.default.svc.cluster.local.

This is a lot of headache infrastructure talk, isn't it? All of these considerations now have to go into your Spring Boot workload? Seems a bit of an "infrastructure code smell".

Luckily, the Spring team has some solutions in mind.

For the Config client part, there is a now a Kubernetes Configuration Watcher, which is essentially a service that... well "watches" the API server for ConfigMap changes based on pre-defined labels. Only this instance would require permissions and NetworkPolicy access to the API server, thus significantly reducing the policy configuration surface area. This watcher can be deployed by a Kubernetes administrator to a different namespace without the developers or the Spring Boot workload instances knowing much about the Kubernetes topology. The Config Watcher is now acting more like a standard Kubernetes operator in that regard, which is more fitting for production environments. The watcher will track all the ConfigMaps or Secrets with a particular label (spring.cloud.kubernetes.config=true or spring.cloud.kubernetes.secret=true respectively), and if any of them changes, it will call the /actuator/refresh endpoint for each one of the pods that have this ConfigMap mounted as a volume. This means you get automatic refresh of the Spring Context even over HTTP without having to resort to Spring Cloud Bus backed by a message broker - a much more simplified approach. Although, if your code already depends on Spring Cloud Bus, that option is supported as well.

If you still want to use Spring Cloud Config Server, for example since you have a complex setup of configuration in a git repo or Vault, you can deploy a Spring Cloud Config Server to Kubernetes directly. In this use case, You will get the added benefit of being able to track both git repos and ConfigMaps/Secrets. Spring Cloud Kubernetes Config Server will also need permissions and NetworkPolicy access to the Kubernetes API server.

What about Spring Cloud Discovery Client? There's a solution for that as well. Spring Cloud Kubernetes Discovery Server is a drop-in replacement for the Eureka API. Existing applications which use Spring Cloud Discovery Client can talk to this server as if it was Spring Cloud Eureka. Unlike Eureka though, they do not need to register themselves with this server, because Spring Cloud Kubernetes Discovery Server gets the service information on available Kubernetes services directly from the Kubernetes API server. Just like Spring Cloud Config Watcher and Spring Cloud Kubernetes Config Server, Spring Cloud Kubernetes Discovery Server requires permissions and NetworkPolicy access to the Kubernetes API server:

Opt for Kubernetes-native service discovery and configuration management. For brownfield applications, use Spring Cloud Kubernetes as a drop-in replacement.

Profiling your Spring Boot application

Throughout this article, I mentioned that you need to profile your application in order to better understand how to configure it for containers. Each application is different and has different requirements in terms of multi-threading and memory, and these settings have a tremendous impact on how your application will perform.

A deep dive on Spring application profiling is out of scope for this document, but here are some pointers you can use as a starting point:

1. Consider testing your app independently of the container constraints first. Just run the JVM on a relatively powerful machine with a healthy amount of CPU and Memory. To set the desired number of cores your JVM will use, set the following command (and adjust the count as needed for different performance testing):

   > java -XX:ActiveProcessorCount=4 target/spring-on-k8s.jar
   

Remember, this setting is quite important since we don't want to let the container "guess" the correct value. It's best to be explicit, especially since other factors use this value for reference - such as the selected garbage collector and the size of your thread pools.

1. The JVM uses a combination of heap memory and non-heap memory. The sum of these two values will have an impact on your container configuration. In order to profile your native memory, set the following JVM arguments:

   java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -jar target/myapp.jar
   

   Native memory usage can be a hidden trojan horse in your memory footprint. You could have a good ratio between heap memory and your container memory, but native memory usage could be significant based on your use case, and consume all the remaining available memory. When profiling with the arguments above, the JVM will begin tracking native memory usage in your application, and will print a usage summary when the application is stopped:

    Native Memory Tracking:
    Total: reserved=10376420413, committed=245521469
    -                 Java Heap (reserved=8589934592, committed=88080384)
                                (mmap: reserved=8589934592, committed=88080384)
    ...
    -    Native Memory Tracking (reserved=3535360, committed=3535360)
                                (malloc=6016 #84)
                                (tracking overhead=3529344)
    ...
   

2. During profiling, include the spring-boot-starter-actuator dependency to expose production-ready endpoints that will help you have a better understanding about your application. For profiling, you'd want access to all endpoints, so set the following property during profiling, but do not keep it in production since it can pose a security risk:

   management.endpoints.web.exposure.include=*
   

   If you're monitoring with JMX, the endpoints are already exposed via the following default setting:

   management.endpoints.jmx.exposure.include=*
   

   The following endpoints will help you out with profiling:

   • /actuator/heapdump will return an hprof dump file that you can load into tools such as JProfiler, the InteliJ Profiler or others.
   • /actuator/threaddump will return a thread dump that you can load into tools such as Fast Thread or jstack.
   • /actuator/metrics will return metrics on the JVM that can be very useful. For example, /actuator/metrics/jvm.classes.loaded will return the number of classes loaded by the JVM.
   • /actuator/prometheus requires an additional dependency on micrometer-registry-prometheus, and will expose the same metrics, but in a format that can be scraped by Prometheus and later presented in Grafana dashboards.
   • Online SaaS services such as DataDog or Tanzu Observability can also provide valuable input.

3. If you have the license, consider using Oracle's Flight Recorder diagnostic and profiling tool. Spring has built-in support for Flight Recorder, by booting your Spring App (2.4 and above) with the following main method:

   @SpringBootApplication
   @ConfigurationPropertiesScan
   public class DemoApplication {
    public static void main(String[] args) {
        SpringApplication springApplication = new SpringApplication(DemoApplication.class);
        springApplication.setApplicationStartup(new FlightRecorderApplicationStartup());
        springApplication.run(args);
    }
   }
   

   And the following JVM arguments on startup:

   java -XX:+UnlockCommercialFeatures -XX:StartFlightRecording:filename=recording.jfr,duration=10s -jar target/myapp.jar
   

   You can instead use a BufferingApplicationStartup as follows:

   @SpringBootApplication
   @ConfigurationPropertiesScan
   public class DemoApplication {
    public static void main(String[] args) {
        SpringApplication springApplication = new SpringApplication(DemoApplication.class);
        springApplication.setApplicationStartup(new BufferingApplicationStartup(4096));
        springApplication.run(args);
    }
   }
   

   The information collected by this solution will be exposed as a json via the actuator endpoint /actuator/startup.

4. Consider using services that offer smart code optimization and recommendations such as Github copilot. If your Spring application uses JPA and Hibernate - consider using Hypersistence Optimizer to optimize your JPA code path. Any code-level improvement means less wasted CPU cycles in production!

5. Test against "real" backing services to better simulate how your code will behave in production. In-memory databases such as H2 and HSQLDB are nice, but with tools like Testcontainers becoming so powerful, H2 and HSQLDB are just becoming technical debt since you're testing against an emulated database when you could have tested against the real thing.

Looking further

Everything I described in this article is relevant for JVM applications at the end of 2022, with Spring Boot 2.x, which is probably the vast majority of the Spring workloads out there.

At SpringOne 2022, the Spring team will release Spring Framework 6.0 and Spring Boot 3.0. The main feature of this release is a requirement for Java 17 or later, and GA support for GraalVM with Spring Native. Spring Native compiles your Java code Ahead of Time. The Java Virtual Machine is replaced by a Substrate VM, which is specific to the target OS and architecture. Your container will require significantly less memory - probably less than 100 MB compared to 2 GB with a JVM. It also means the container starts up almost immediately - we're now talking about milliseconds instead of seconds. This makes Spring-based applications an ideal candidate for serverless solutions such as Knative and AWS Lambda, since these solutions tend to scale to zero and need immediate response time from a cold start.

Clearly, such a meaningful change becomes a different discussion all-together, and would require revisiting most of the recommendations in this article in the future, perhaps in a separate article.

Although Spring Native shows a lot of promise, the JVM isn't going away anytime soon. GraalVM-based solutions have tradeoffs as well:

• They boot a lot faster, but don't achieve the same peak performance.
• You cannot use Java agents and some instrumentation solutions such as those used by some observability solutions.
• There is still work to be done in the third-party frameworks community to adapt the framework to this new architecture.
• Compilation time is longer, although it has improved considerably.

Another interesting option that could gain traction in the future is the use of JLink. This process allows you to selectively choose only the JRE modules that are required to run your application. It can greatly reduce the container size and require less runtime memory. Unlike Spring Native - this will still result in a JVM runtime, just a more lightweight version of it. As mentioned in this very old Github Issue, the focus appears to be shifting towards Spring Native, and there are no plans to focus more work on the complementary Project Jigsaw which was meant to automate the module building for frameworks such as Spring. Still, you can use JLink directly as demoed here by Dr. Dave Syer and by Adib Saikali. Compared to the images built by the Paketo Buildpacks, the JLink image can be around 40% smaller. One of the challenges for Spring is the need to still include the java.desktop module, which is a very large module and doesn't seem right for a server-side Spring application. In Spring Framework 6.0, this module would not be needed anymore and would result in even smaller images.

A different perspective on this problem domain is Project Leyden, which was also mentioned as one of the paths forward for the Spring Framework. Leyden also aims to provide faster startup times and reduced memory footprint compared to a full-fledged JVM, using the concept of static images - basically a finite and closed version of the JVM for your specific application only. If the JDK is a Word document, Project Leyden is the PDF version of it.

Testing my own recommendations

I set myself a challenge when I began working on this article. Tanzu Application Platform is a Platform-as-a-Service that runs on any Kubernetes. You as a developer provide a simple workload yaml file containing a reference to the git repo with your source code, and in the end you have a working application in Kubernetes, powered by Knative runtime and automatic scale-to-zero. 12 lines of yaml generate thousands of lines of yaml that your team doesn't have to maintain themselves.

Since the platform is developed by the same team that develops the Spring framework, it has production-ready defaults and best practices applied to it based on years of research and expertise. I thought it would be a good idea to compare my recommendations in this article with the resulting Knative service and pods generated by the platform. So, without further ado, here's the a sample Kubernetes pod, created by a Knative service, which was created by Tanzu Application Platform (some lines removed for readability):

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: tanzu-java-web-app
  ...
spec:
  template:
    metadata:
      annotations:
      ...
      labels:
      ...
    spec:
      containers:
      - env:
        - name: JAVA_TOOL_OPTIONS
          value: -Dmanagement.endpoint.health.probes.add-additional-paths="true" -Dmanagement.endpoint.health.show-details=always -Dmanagement.endpoints.web.base-path="/actuator" -Dmanagement.endpoints.web.exposure.include=* -Dmanagement.health.probes.enabled="true" -Dmanagement.server.port="8081" -Dserver.port="8080" -Dserver.shutdown.grace-period="24s"
        image: odedia/tanzu-java-web-app-demos@sha256:6884b8289e21b1677b91aebd40a3198d07a6951fb912efaee504945b105b682b
        livenessProbe:
          httpGet:
            path: /livez
            port: 8080
            scheme: HTTP
        name: workload
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
            scheme: HTTP
        resources: {}
        securityContext:
          runAsUser: 1000
      serviceAccountName: default

Here is a corresponding pod that was created by the above Knative service. The pod also has a Knative sidecar container which is not relevant for our use case, so I removed it for readability along with other labels and annotations:

apiVersion: v1
kind: Pod
metadata:
  annotations:
  ...
  labels:
  ...
  name: tanzu-java-web-app-00001-deployment-7946d647f9-5gvlf
  namespace: demos
spec:
  containers:
  - env:
    - name: JAVA_TOOL_OPTIONS
      value: -Dmanagement.endpoint.health.probes.add-additional-paths="true" -Dmanagement.endpoint.health.show-details=always
        -Dmanagement.endpoints.web.base-path="/actuator" -Dmanagement.endpoints.web.exposure.include=*
        -Dmanagement.health.probes.enabled="true" -Dmanagement.server.port="8081"
        -Dserver.port="8080" -Dserver.shutdown.grace-period="24s"
    - name: PORT
      value: "8080"
    image: harbor.mysite.com/app/tanzu-java-web-app-demos@sha256:d76d166baa827a998a4637c2771af72b1ea33240ed15107151da99b7a8eb7e39
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        httpGet:
          path: /wait-for-drain
          port: 8022
          scheme: HTTP
    livenessProbe:
      failureThreshold: 3
      httpGet:
        httpHeaders:
        - name: K-Kubelet-Probe
          value: queue
        path: /livez
        port: 8080
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: workload
    ports:
    - containerPort: 8080
      name: user-port
      protocol: TCP
    resources: {}
    securityContext:
      runAsUser: 1000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
  dnsPolicy: ClusterFirst
  enableServiceLinks: false
  imagePullSecrets:
  - name: registry-credentials
  - name: tap-registry
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  ...

This seems very close to the conclusions in this article. Let's highlight the applied JAVA_TOOL_OPTIONS parameters:

-Dmanagement.endpoint.health.probes.add-additional-paths="true" 
-Dmanagement.endpoint.health.show-details=always 
-Dmanagement.endpoints.web.base-path="/actuator" 
-Dmanagement.endpoints.web.exposure.include=* 
-Dmanagement.health.probes.enabled="true" 
-Dmanagement.server.port="8081" 
-Dserver.port="8080" 
-Dserver.shutdown.grace-period="24s"

You might be wondering: "where are the memory parameters? Where is the configuration for ActiveProcessorCount"? These are applied by the Java Memory Calculator on startup. Nirvana :). Here's the startup logs of the pod listed above. Notice the memory calculations applied:

> kubectl logs tanzu-java-web-app-00001-deployment-7f89bf96fc-cxbtb
Defaulted container "workload" out of: workload, queue-proxy
Setting Active Processor Count to 32
Calculating JVM memory based on 110224100K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx109626290K -XX:MaxMetaspaceSize=85809K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 110224100K, Thread Count: 250, Loaded Class Count: 12736, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 127 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Dmanagement.endpoint.health.probes.add-additional-paths="true" -Dmanagement.endpoint.health.show-details=always -Dmanagement.endpoints.web.base-path="/actuator" -Dmanagement.endpoints.web.exposure.include=* -Dmanagement.health.probes.enabled="true" -Dmanagement.server.port="8081" -Dserver.port="8080" -Dserver.shutdown.grace-period="24s" -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=32 -XX:MaxDirectMemorySize=10M -Xmx109626290K -XX:MaxMetaspaceSize=85809K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.4)

2022-10-11 09:12:21.357  INFO 1 --- [           main] com.example.springboot.Application       : Starting Application v0.0.1-SNAPSHOT using Java 11.0.16.1 on tanzu-java-web-app-00001-deployment-7f89bf96fc-cxbtb with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
...

The only thing I seemed to have missed is related to security: It's always advisable to run your app with a service account and a non-root security context:

    securityContext:
      runAsUser: 1000
  ...
  serviceAccount: default

Conclusion

In this article, I highlighted some of the challenges and considerations that developers and Devops teams need to take in to account when deploying Spring applications to Kubernetes. There are of-course many aspects that I haven't mentioned, so I would love to hear your feedback in the comments section. What tips can you share with the community? How can we achieve peak performance, consistency and reliability in production for our Spring workloads?

Thank you for reading.

Some content is based on "Effective Spring" by Adib Saikali. Additional feedback provided by Stuart Charlton, Dr. David Syer and Ryan Baxter.

If you’re an overcast.fm fan like me, you can listen to it on overcast with this link:

https://overcast.fm/+1LYCSCYYE

Cloud Native Buildpacks is a CNCF specification that "helps you transform your application source code into images that can run on any cloud".

In simpler terms, it is an automated source-to-OCI-image solution that is designed to take the toil of writing Dockerfiles from the hands of developers or devops teams. More importantly - it allows for proactive CVE patching. Since CNB is responsible for building the image, it knows how that image was built - its various layers, including the underlying base OS, which is called a stack, as well as runtime layers such as OpenJDK, .NET Core runtime etc.

Working with our customers at VMware Tanzu, I began to notice some patterns around making sure CNB work well in the "real world" - that is, with real customer codebases, airgapped environments, and multi-module repositories. These are usually much different than your typical "Hello World" demo apps, so I figured I'd list some of the caveats and best practices that I find for the most common languages.

Choosing a platform

Cloud Native Buildpacks is a specification. It does not dictate the target runtime. As a result - there are various platforms it can run on, such as:

• Spring mvn/gradle plugin
• pack cli for local OCI image building
• Azure Spring Cloud
• Google Cloud
• kpack as an open source automated image building service on Kubernetes
• Tanzu Build Service - a commercial, enterprise supported distribution based on kpack

For this article, I'm going to use Tanzu Build Service since that is what I'm most comfortable with and also because it's probably the most feature-rich distribution. It's also included as part of Tanzu Application Platform . Since Tanzu Build Service is based on kpack, almost everything can be applied to kpack as well.

Buildpacks in review

Spring Boot "hello world!"

Let's start with a simple Spring Boot hello world app just to get into the mood:

Git repo

The repository is available at https://github.com/odedia/spring-on-k8s.git

Build command

This is as straigntforward as it gets. Our repository is based on maven, and Tanzu Build Service detects it automatically since a pom.xml is available at the root of the repository. The command to build the image is:

> kp image save spring-hello-world \
    --tag harbor.registry.com/apps/spring-hello-world \
    --git https://github.com/odedia/spring-on-k8s.git \
    --git-revision master

Build logs output

Let's check the build logs:

> kp build logs spring-hello-world
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  - source: {}
  + source:
  +   git:
  +     revision: 50c46f40baa6ac8a72498d516624fc78d95f0d2a
  +     url: https://github.com/odedia/spring-on-k8s.git
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/odedia/spring-on-k8s.git" @ "50c46f40baa6ac8a72498d516624fc78d95f0d2a"...
Successfully cloned "https://github.com/odedia/spring-on-k8s.git" @ "50c46f40baa6ac8a72498d516624fc78d95f0d2a" in path "/workspace"
===> ANALYZE
===> DETECT
8 of 35 buildpacks participating
paketo-buildpacks/ca-certificates   3.0.2
paketo-buildpacks/bellsoft-liberica 9.0.2
paketo-buildpacks/syft              1.3.1
paketo-buildpacks/maven             6.0.2
paketo-buildpacks/executable-jar    6.0.2
paketo-buildpacks/apache-tomcat     7.0.3
paketo-buildpacks/dist-zip          5.0.2
paketo-buildpacks/spring-boot       5.3.0
===> RESTORE
Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:java-security-properties" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jre" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:helper" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Reusing cached layer

Paketo BellSoft Liberica Buildpack 9.0.2
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_TYPE                 JRE             the JVM type - JDK or JRE
    $BP_JVM_VERSION              11              the Java version
  Launch Configuration:
    $BPL_DEBUG_ENABLED           false           enables Java remote debugging support
    $BPL_DEBUG_PORT              8000            configure the remote debugging port
    $BPL_DEBUG_SUSPEND           false           configure whether to suspend execution until a debugger has attached
    $BPL_HEAP_DUMP_PATH                          write heap dumps on error to this path
    $BPL_JAVA_NMT_ENABLED        true            enables Java Native Memory Tracking (NMT)
    $BPL_JAVA_NMT_LEVEL          summary         configure level of NMT, summary or detail
    $BPL_JFR_ARGS                                configure custom Java Flight Recording (JFR) arguments
    $BPL_JFR_ENABLED             false           enables Java Flight Recording (JFR)
    $BPL_JMX_ENABLED             false           enables Java Management Extensions (JMX)
    $BPL_JMX_PORT                5000            configure the JMX port
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags
  BellSoft Liberica JDK 11.0.13: Contributing to layer
    Reusing cached download from buildpack
    Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jdk
    Adding 128 container CA certificates to JVM truststore
    Writing env.build/JAVA_HOME.override
    Writing env.build/JDK_HOME.override
  BellSoft Liberica JRE 11.0.13: Reusing cached layer
  Launch Helper: Reusing cached layer
  Java Security Properties: Reusing cached layer

Paketo Syft Buildpack 1.3.1
  https://github.com/paketo-buildpacks/syft
    Reusing cached download from buildpack
    Writing env.build/SYFT_CHECK_FOR_APP_UPDATE.default

Paketo Maven Buildpack 6.0.2
  https://github.com/paketo-buildpacks/maven
  Build Configuration:
    $BP_MAVEN_BUILD_ARGUMENTS  -Dmaven.test.skip=true package  the arguments to pass to Maven
    $BP_MAVEN_BUILT_ARTIFACT   target/*.[ejw]ar                the built application artifact explicitly.  Supersedes $BP_MAVEN_BUILT_MODULE
    $BP_MAVEN_BUILT_MODULE                                     the module to find application artifact in
    $BP_MAVEN_POM_FILE         pom.xml                         the location of the main pom.xml file, relative to the application root
    Creating cache directory /home/cnb/.m2
  Compiled Application: Contributing to layer
    Executing mvnw --batch-mode -Dmaven.test.skip=true package
      [INFO] Scanning for projects...
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.4.3/spring-boot-starter-parent-2.4.3.pom
...
...
...
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.4/org.eclipse.sisu.inject-0.3.4.jar (379 kB at 2.9 MB/s)
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/eclipse/sisu/org.eclipse.sisu.plexus/0.3.4/org.eclipse.sisu.plexus-0.3.4.jar (205 kB at 1.5 MB/s)
      [INFO] Replacing main artifact with repackaged archive
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  10.232 s
      [INFO] Finished at: 2022-01-16T10:02:48Z
      [INFO] ------------------------------------------------------------------------
  Removing source code

Paketo Executable JAR Buildpack 6.0.2
  https://github.com/paketo-buildpacks/executable-jar
  Class Path: Contributing to layer
    Writing env/CLASSPATH.delim
    Writing env/CLASSPATH.prepend
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
    task:           java org.springframework.boot.loader.JarLauncher (direct)
    web:            java org.springframework.boot.loader.JarLauncher (direct)

Paketo Spring Boot Buildpack 5.3.0
  https://github.com/paketo-buildpacks/spring-boot
  Creating slices from layers index
    dependencies
    spring-boot-loader
    snapshot-dependencies
    application
  Launch Helper: Reusing cached layer
  Spring Cloud Bindings 1.8.0: Reusing cached layer
  Web Application Type: Reusing cached layer
  4 application slices
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.version
===> EXPORT
Reusing layers from image 'harbor.registry.com/apps/spring-hello-world@sha256:e135f066c71524c6dc8343d4c803c417240c1ffc1563daa0224c865e45900d53'
Reusing layer 'paketo-buildpacks/ca-certificates:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:jre'
Reusing layer 'paketo-buildpacks/executable-jar:classpath'
Reusing layer 'paketo-buildpacks/spring-boot:helper'
Reusing layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Reusing layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding layer 'launch.sbom'
Reusing 5/5 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving harbor.registry.com/apps/spring-hello-world...
*** Images (sha256:792fee75f5627daed9e8731971d134a28112cc952359d983ea901fca5cf7f522):
      harbor.registry.com/apps/spring-hello-world
      harbor.registry.com/apps/spring-hello-world:b1.20220116.100212
Adding cache layer 'paketo-buildpacks/bellsoft-liberica:jdk'
Adding cache layer 'paketo-buildpacks/syft:syft'
Adding cache layer 'paketo-buildpacks/maven:application'
Adding cache layer 'paketo-buildpacks/maven:cache'
Adding cache layer 'cache.sbom'
===> COMPLETION
Build successful

Running the image

Let's see if our OCI image runs as expected:

> docker run -p 8080:8080 harbor.registry.com/apps/spring-hello-world:b1.20220116.100212

Unable to find image 'harbor.registry.com/apps/spring-hello-world:b1.20220116.100212' locally
b1.20220116.100212: Pulling from apps/spring-hello-world
bf99a8b93828: Already exists
1da1131b1360: Pull complete
48bde579d8cc: Pull complete
1929d0623312: Pull complete
1fea93ec33be: Pull complete
fed2bcb93d83: Pull complete
d06e3ac07eba: Pull complete
a8090bf3b91d: Pull complete
204c9b3ef04c: Pull complete
95d9537f4e2e: Pull complete
40cbb147917e: Pull complete
b8e5cf599c18: Pull complete
bd74081a3b16: Pull complete
1d17a9810ba3: Pull complete
be14b062b309: Pull complete
33a1d27f7a3f: Pull complete
89732bc75041: Pull complete
96580d31d7e3: Pull complete
40d17e1b116e: Pull complete
02d76976a806: Pull complete
b39b18afe74e: Pull complete
Digest: sha256:792fee75f5627daed9e8731971d134a28112cc952359d983ea901fca5cf7f522
Status: Downloaded newer image for harbor.registry.com/apps/spring-hello-world:b1.20220116.100212
Setting Active Processor Count to 4
WARNING: Unable to convert memory limit "max" from path "/sys/fs/cgroup/memory.max" as int: memory size "max" does not match pattern "^([\\d]+)([kmgtKMGT]?)$"
Calculating JVM memory based on 1284376K available memory
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx687495K -XX:MaxMetaspaceSize=84880K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1284376K, Thread Count: 250, Loaded Class Count: 12572, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 128 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx687495K -XX:MaxMetaspaceSize=84880K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.4.3)

2022-01-16 10:08:34.050  INFO 1 --- [           main] c.vmware.demos.springonk8s.Application   : Starting Application v1.0.0-SNAPSHOT using Java 11.0.13 on 3b4ec7e4ed92 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2022-01-16 10:08:34.055  INFO 1 --- [           main] c.vmware.demos.springonk8s.Application   : No active profile set, falling back to default profiles: default
2022-01-16 10:08:35.541  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2022-01-16 10:08:35.560  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-01-16 10:08:35.560  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.43]
2022-01-16 10:08:35.640  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-01-16 10:08:35.641  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1499 ms
2022-01-16 10:08:36.079  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2022-01-16 10:08:36.433  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2022-01-16 10:08:36.494  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2022-01-16 10:08:36.517  INFO 1 --- [           main] c.vmware.demos.springonk8s.Application   : Started Application in 3.032 seconds (JVM running for 3.479)
2022-01-16 10:08:42.024  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-01-16 10:08:42.026  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2022-01-16 10:08:42.028  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms

Let's check the application:

Looking good! Let's move to a more robust application.

Spring Boot Pet Clinic backend application

This is the classic demo application for Spring Boot. I'm going to use a version of this app that only runs the backend, since I find it quite rare to run HTML+Javascript in a Spring Boot container - you'd usually run an Angular/React/VueVS application in a static code server such as NGINX. For now, lets build our REST APIs backend:

Git Repo

This application is located in https://github.com/spring-petclinic/spring-petclinic-rest.git

Build command

The Java Buildpack (and Spring), is probably the most robust, so building the OCI image is still quite straightforward. Just point to the git repo:

> kp image save spring-petclinic-backend \
    --tag harbor.registry.com/apps/spring-petclinic-backend \
    --git https://github.com/spring-petclinic/spring-petclinic-rest.git \
    --git-revision master
Creating Image Resource...
Image Resource "spring-petclinic-backend" created

Build logs output

> kp image save spring-petclinic-backend \
    --tag harbor.registry.com/apps/spring-petclinic-backend \
    --git https://github.com/spring-petclinic/spring-petclinic-rest.git \
    --git-revision master
Creating Image Resource...
Image Resource "spring-petclinic-backend" created
(⎈ |tap:default)➜  ~
> kp build logs spring-petclinic-backend
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  - source: {}
  + source:
  +   git:
  +     revision: 8da34240a5a9a04809f714a76cee82c6e27bea14
  +     url: https://github.com/spring-petclinic/spring-petclinic-rest.git
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/spring-petclinic/spring-petclinic-rest.git" @ "8da34240a5a9a04809f714a76cee82c6e27bea14"...
Successfully cloned "https://github.com/spring-petclinic/spring-petclinic-rest.git" @ "8da34240a5a9a04809f714a76cee82c6e27bea14" in path "/workspace"
===> ANALYZE
Previous image with name "harbor.registry.com/apps/spring-petclinic-backend" not found
===> DETECT
8 of 35 buildpacks participating
paketo-buildpacks/ca-certificates   3.0.2
paketo-buildpacks/bellsoft-liberica 9.0.2
paketo-buildpacks/syft              1.3.1
paketo-buildpacks/maven             6.0.2
paketo-buildpacks/executable-jar    6.0.2
paketo-buildpacks/apache-tomcat     7.0.3
paketo-buildpacks/dist-zip          5.0.2
paketo-buildpacks/spring-boot       5.3.0
===> RESTORE
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper

Paketo BellSoft Liberica Buildpack 9.0.2
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_TYPE                 JRE             the JVM type - JDK or JRE
    $BP_JVM_VERSION              11              the Java version
  Launch Configuration:
    $BPL_DEBUG_ENABLED           false           enables Java remote debugging support
    $BPL_DEBUG_PORT              8000            configure the remote debugging port
    $BPL_DEBUG_SUSPEND           false           configure whether to suspend execution until a debugger has attached
    $BPL_HEAP_DUMP_PATH                          write heap dumps on error to this path
    $BPL_JAVA_NMT_ENABLED        true            enables Java Native Memory Tracking (NMT)
    $BPL_JAVA_NMT_LEVEL          summary         configure level of NMT, summary or detail
    $BPL_JFR_ARGS                                configure custom Java Flight Recording (JFR) arguments
    $BPL_JFR_ENABLED             false           enables Java Flight Recording (JFR)
    $BPL_JMX_ENABLED             false           enables Java Management Extensions (JMX)
    $BPL_JMX_PORT                5000            configure the JMX port
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags
  BellSoft Liberica JDK 11.0.13: Contributing to layer
    Reusing cached download from buildpack
    Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jdk
    Adding 128 container CA certificates to JVM truststore
    Writing env.build/JAVA_HOME.override
    Writing env.build/JDK_HOME.override
  BellSoft Liberica JRE 11.0.13: Contributing to layer
    Reusing cached download from buildpack
    Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
    Adding 128 container CA certificates to JVM truststore
    Writing env.launch/BPI_APPLICATION_PATH.default
    Writing env.launch/BPI_JVM_CACERTS.default
    Writing env.launch/BPI_JVM_CLASS_COUNT.default
    Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
    Writing env.launch/JAVA_HOME.default
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim
    Writing env.launch/MALLOC_ARENA_MAX.default
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/active-processor-count
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/java-opts
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/jvm-heap
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/link-local-dns
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-configurer
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/jmx
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/jfr
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/nmt
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-classpath-9
    Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/debug-9
  Java Security Properties: Contributing to layer
    Writing env.launch/JAVA_SECURITY_PROPERTIES.default
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim

Paketo Syft Buildpack 1.3.1
  https://github.com/paketo-buildpacks/syft
    Reusing cached download from buildpack
    Writing env.build/SYFT_CHECK_FOR_APP_UPDATE.default

Paketo Maven Buildpack 6.0.2
  https://github.com/paketo-buildpacks/maven
  Build Configuration:
    $BP_MAVEN_BUILD_ARGUMENTS  -Dmaven.test.skip=true package  the arguments to pass to Maven
    $BP_MAVEN_BUILT_ARTIFACT   target/*.[ejw]ar                the built application artifact explicitly.  Supersedes $BP_MAVEN_BUILT_MODULE
    $BP_MAVEN_BUILT_MODULE                                     the module to find application artifact in
    $BP_MAVEN_POM_FILE         pom.xml                         the location of the main pom.xml file, relative to the application root
    Creating cache directory /home/cnb/.m2
  Compiled Application: Contributing to layer
    Executing mvnw --batch-mode -Dmaven.test.skip=true package
      [INFO] Scanning for projects...
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.6.2/spring-boot-starter-parent-2.6.2.pom
...
...
...
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.3.0/plexus-utils-3.3.0.jar (263 kB at 16 MB/s)
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.19/commons-compress-1.19.jar (615 kB at 23 MB/s)
      [INFO] Building jar: /workspace/target/spring-petclinic-rest-2.6.2.jar
      [INFO]
      [INFO] --- spring-boot-maven-plugin:2.6.2:repackage (repackage) @ spring-petclinic-rest ---
      [INFO] Replacing main artifact with repackaged archive
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  23.587 s
      [INFO] Finished at: 2022-01-16T10:31:48Z
      [INFO] ------------------------------------------------------------------------
  Removing source code

Paketo Executable JAR Buildpack 6.0.2
  https://github.com/paketo-buildpacks/executable-jar
  Class Path: Contributing to layer
    Writing env/CLASSPATH.delim
    Writing env/CLASSPATH.prepend
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
    task:           java org.springframework.boot.loader.JarLauncher (direct)
    web:            java org.springframework.boot.loader.JarLauncher (direct)

Paketo Spring Boot Buildpack 5.3.0
  https://github.com/paketo-buildpacks/spring-boot
  Creating slices from layers index
    dependencies
    spring-boot-loader
    snapshot-dependencies
    application
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
  Spring Cloud Bindings 1.8.0: Contributing to layer
    Reusing cached download from buildpack
    Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
  Web Application Type: Contributing to layer
    Servlet web application detected
    Writing env.launch/BPL_JVM_THREAD_COUNT.default
  4 application slices
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.version
===> EXPORT
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'paketo-buildpacks/bellsoft-liberica:helper'
Adding layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'
Adding layer 'paketo-buildpacks/executable-jar:classpath'
Adding layer 'paketo-buildpacks/spring-boot:helper'
Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding layer 'launch.sbom'
Adding 5/5 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving harbor.registry.com/apps/spring-petclinic-backend...
*** Images (sha256:6cf16c4573c95f5e66e40740d9f6238377b3aaab38176e7986310c21bdd6094a):
      harbor.registry.com/apps/spring-petclinic-backend
      harbor.registry.com/apps/spring-petclinic-backend:b1.20220116.103057
Adding cache layer 'paketo-buildpacks/bellsoft-liberica:jdk'
Adding cache layer 'paketo-buildpacks/syft:syft'
Adding cache layer 'paketo-buildpacks/maven:application'
Adding cache layer 'paketo-buildpacks/maven:cache'
Adding cache layer 'cache.sbom'
===> COMPLETION
Build successful

Running the image

> docker run -p 8080:8080 harbor.registry.com/apps/spring-petclinic-backend:b1.20220116.103057
Unable to find image 'harbor.registry.com/apps/spring-petclinic-backend:b1.20220116.103057' locally
b1.20220116.103057: Pulling from apps/spring-petclinic-backend
bf99a8b93828: Already exists
1da1131b1360: Already exists
48bde579d8cc: Already exists
1929d0623312: Already exists
1fea93ec33be: Already exists
fed2bcb93d83: Already exists
d06e3ac07eba: Already exists
a8090bf3b91d: Already exists
204c9b3ef04c: Already exists
95d9537f4e2e: Already exists
40cbb147917e: Already exists
b8e5cf599c18: Already exists
bd74081a3b16: Already exists
50105f4a3af9: Pull complete
b218c24b76da: Pull complete
afd258a43f9e: Pull complete
89732bc75041: Pull complete
428c2ea77ec0: Pull complete
40d17e1b116e: Pull complete
c3e95d93d7ac: Pull complete
b39b18afe74e: Pull complete
Digest: sha256:6cf16c4573c95f5e66e40740d9f6238377b3aaab38176e7986310c21bdd6094a
Status: Downloaded newer image for harbor.registry.com/apps/spring-petclinic-backend:b1.20220116.103057
Setting Active Processor Count to 4
WARNING: Unable to convert memory limit "max" from path "/sys/fs/cgroup/memory.max" as int: memory size "max" does not match pattern "^([\\d]+)([kmgtKMGT]?)$"
Calculating JVM memory based on 1280440K available memory
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx644437K -XX:MaxMetaspaceSize=124002K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1280440K, Thread Count: 250, Loaded Class Count: 19479, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 128 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx644437K -XX:MaxMetaspaceSize=124002K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.2)

INFO  PetClinicApplication - Starting PetClinicApplication v2.6.2 using Java 11.0.13 on 0b6241f0c523 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
DEBUG PetClinicApplication - Running with Spring Boot v2.6.2, Spring v5.3.14
INFO  PetClinicApplication - The following profiles are active: hsqldb,spring-data-jpa
INFO  Version - HV000001: Hibernate Validator 6.2.0.Final
INFO  RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
INFO  RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 97 ms. Found 7 JPA repository interfaces.
INFO  TomcatWebServer - Tomcat initialized with port(s): 9966 (http)
INFO  Http11NioProtocol - Initializing ProtocolHandler ["http-nio-9966"]
INFO  StandardService - Starting service [Tomcat]
INFO  StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.56]
INFO  [/petclinic] - Initializing Spring embedded WebApplicationContext
INFO  ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 2210 ms
INFO  HikariDataSource - HikariPool-1 - Starting...
INFO  PoolBase - HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
INFO  HikariDataSource - HikariPool-1 - Start completed.
INFO  LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
INFO  Version - HHH000412: Hibernate ORM core version 5.6.3.Final
INFO  Version - HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
INFO  Dialect - HHH000400: Using dialect: org.hibernate.dialect.HSQLDialect
INFO  JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
INFO  LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default'
INFO  EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path '/actuator'
INFO  UserDetailsServiceAutoConfiguration -

Using generated security password: 5b765681-8357-403d-8e73-7edc4554cccc

INFO  DefaultSecurityFilterChain - Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@4f1072ee, org.springframework.security.web.context.SecurityContextPersistenceFilter@3fe33c59, org.springframework.security.web.header.HeaderWriterFilter@60eb9c29, org.springframework.security.web.authentication.logout.LogoutFilter@293988dc, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@34e700f4, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@3bbab2c9, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@7b0a69b9, org.springframework.security.web.session.SessionManagementFilter@2f0a29f7, org.springframework.security.web.access.ExceptionTranslationFilter@63019596, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@5a513627]
INFO  Http11NioProtocol - Starting ProtocolHandler ["http-nio-9966"]
INFO  TomcatWebServer - Tomcat started on port(s): 9966 (http) with context path '/petclinic'
INFO  PetClinicApplication - Started PetClinicApplication in 6.817 seconds (JVM running for 7.35)

Let's check the output by running one of the APIs at http://localhost:8080/petclinic/swagger-ui/index.html:

Hmm.... what happened here? If we check the logs above, it seems that our sample code starts at port 9966. It is considered a common practice (but definitely not required) to run user-facing containers such as these on port 8080, and map to whatever port we want outside the container. We could simple run the below command and it would work fine:

> docker run -p 8080:9966 harbor.registry.com/apps/spring-petclinic-backend:b1.20220116.103057

However, I like to have repeatable practices. So, let's fork the repo and fix the server.port setting in application.properties.

Making changes

The forked repo is located at https://github.com/odedia/spring-petclinic-rest. Let's update https://github.com/odedia/spring-petclinic-rest/blob/master/src/main/resources/application.properties and change server.port=9966 to server.port=8080.

To update our build to point to the updated repo, we simply need to update the kp image save command with the updated git URL:

> kp image save spring-petclinic-backend \
    --tag harbor.registry.com/apps/spring-petclinic-backend \
    --git https://github.com/odedia/spring-petclinic-rest.git \
    --git-revision master
Patching Image Resource...
Image Resource "spring-petclinic-backend" patched

If we'll check the build logs now, we'll find that the build is a lot faster and doesn't download all the maven dependencies again. That's because Tanzu Build Service caches layers as needed, and the maven cache layer did not change:

> kp build logs spring-petclinic-backend
===> PREPARE
Build reason(s): COMMIT,CONFIG
COMMIT:
  - 8da34240a5a9a04809f714a76cee82c6e27bea14
  + c31f00af9fae5ef63c323cfbf5f890f252d8669a
CONFIG:
  resources: {}
  source:
    git:
  -     revision: 8da34240a5a9a04809f714a76cee82c6e27bea14
  -     url: https://github.com/spring-petclinic/spring-petclinic-rest.git
  +     revision: c31f00af9fae5ef63c323cfbf5f890f252d8669a
  +     url: https://github.com/odedia/spring-petclinic-rest.git
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/odedia/spring-petclinic-rest.git" @ "c31f00af9fae5ef63c323cfbf5f890f252d8669a"...
Successfully cloned "https://github.com/odedia/spring-petclinic-rest.git" @ "c31f00af9fae5ef63c323cfbf5f890f252d8669a" in path "/workspace"
===> ANALYZE
===> DETECT
8 of 35 buildpacks participating
paketo-buildpacks/ca-certificates   3.0.2
paketo-buildpacks/bellsoft-liberica 9.0.2
paketo-buildpacks/syft              1.3.1
paketo-buildpacks/maven             6.0.2
paketo-buildpacks/executable-jar    6.0.2
paketo-buildpacks/apache-tomcat     7.0.3
paketo-buildpacks/dist-zip          5.0.2
paketo-buildpacks/spring-boot       5.3.0
===> RESTORE
Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:java-security-properties" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jre" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:helper" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jdk" from cache
Restoring metadata for "paketo-buildpacks/syft:syft" from cache
Restoring metadata for "paketo-buildpacks/maven:cache" from cache
Restoring metadata for "paketo-buildpacks/maven:application" from cache
Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
Restoring data for "paketo-buildpacks/bellsoft-liberica:jdk" from cache
Restoring data for "paketo-buildpacks/syft:syft" from cache
Restoring data for "paketo-buildpacks/maven:application" from cache
Restoring data for "paketo-buildpacks/maven:cache" from cache
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Reusing cached layer

Paketo BellSoft Liberica Buildpack 9.0.2
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_TYPE                 JRE             the JVM type - JDK or JRE
    $BP_JVM_VERSION              11              the Java version
  Launch Configuration:
    $BPL_DEBUG_ENABLED           false           enables Java remote debugging support
    $BPL_DEBUG_PORT              8000            configure the remote debugging port
    $BPL_DEBUG_SUSPEND           false           configure whether to suspend execution until a debugger has attached
    $BPL_HEAP_DUMP_PATH                          write heap dumps on error to this path
    $BPL_JAVA_NMT_ENABLED        true            enables Java Native Memory Tracking (NMT)
    $BPL_JAVA_NMT_LEVEL          summary         configure level of NMT, summary or detail
    $BPL_JFR_ARGS                                configure custom Java Flight Recording (JFR) arguments
    $BPL_JFR_ENABLED             false           enables Java Flight Recording (JFR)
    $BPL_JMX_ENABLED             false           enables Java Management Extensions (JMX)
    $BPL_JMX_PORT                5000            configure the JMX port
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags
  BellSoft Liberica JDK 11.0.13: Reusing cached layer
  BellSoft Liberica JRE 11.0.13: Reusing cached layer
  Launch Helper: Reusing cached layer
  Java Security Properties: Reusing cached layer

Paketo Syft Buildpack 1.3.1
  https://github.com/paketo-buildpacks/syft

Paketo Maven Buildpack 6.0.2
  https://github.com/paketo-buildpacks/maven
  Build Configuration:
    $BP_MAVEN_BUILD_ARGUMENTS  -Dmaven.test.skip=true package  the arguments to pass to Maven
    $BP_MAVEN_BUILT_ARTIFACT   target/*.[ejw]ar                the built application artifact explicitly.  Supersedes $BP_MAVEN_BUILT_MODULE
    $BP_MAVEN_BUILT_MODULE                                     the module to find application artifact in
    $BP_MAVEN_POM_FILE         pom.xml                         the location of the main pom.xml file, relative to the application root
    Creating cache directory /home/cnb/.m2
  Compiled Application: Contributing to layer
    Executing mvnw --batch-mode -Dmaven.test.skip=true package
      [INFO] Scanning for projects...
      [INFO]
      [INFO] ---------< org.springframework.samples:spring-petclinic-rest >----------
      [INFO] Building spring-petclinic-rest 2.6.2
      [INFO] --------------------------------[ jar ]---------------------------------
      [INFO]
      [INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (default) @ spring-petclinic-rest ---
      [INFO] argLine set to -javaagent:/home/cnb/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/workspace/target/jacoco.exec,excludes=**/org/springframework/samples/petclinic/rest/dto/**:**/org/springframework/samples/petclinic/rest/api/**
      [INFO]
      [INFO] --- openapi-generator-maven-plugin:5.2.1:generate (default) @ spring-petclinic-rest ---
      [INFO] Generating with dryRun=false
      [INFO] Output directory (/workspace/target/generated-sources/openapi) does not exist, or is inaccessible. No file (.openapi-generator-ignore) will be evaluated.
      [INFO] OpenAPI Generator: spring (server)
      [INFO] Generator 'spring' is considered stable.
      [INFO] ----------------------------------
      [WARNING] java8 option has been deprecated as it's set to true by default (JDK7 support has been deprecated)
      [INFO] Environment variable JAVA_POST_PROCESS_FILE not defined so the Java code may not be properly formatted. To define it, try 'export JAVA_POST_PROCESS_FILE="/usr/local/bin/clang-format -i"' (Linux/Mac)
      [INFO] NOTE: To enable file post-processing, 'enablePostProcessFile' must be set to `true` (--enable-post-process-file for CLI).
      [INFO] Invoker Package Name, originally not set, is now derived from api package name: org.springframework.samples.petclinic.rest
      [INFO] Processing operation failingRequest
      [INFO] Processing operation listOwners
      [INFO] Processing operation addOwner
      [INFO] Processing operation getOwner
      [INFO] Processing operation updateOwner
      [INFO] Processing operation addPet
      [INFO] Processing operation getPet
      [INFO] Processing operation updatePet
      [INFO] Processing operation addVisit
      [INFO] Processing operation listPetTypes
      [INFO] Processing operation listVets
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/OwnerAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/OwnerDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/OwnerFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetTypeAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetTypeDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetTypeFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/RestErrorDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/RoleDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/SpecialtyDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/UserDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/ValidationMessageDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VetAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VetDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VetFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VisitAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VisitDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VisitFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/OopsApi.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/OwnersApi.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/PettypesApi.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/VetsApi.java
      [INFO] Skipped /workspace/target/generated-sources/openapi/pom.xml (Skipped by supportingFiles options supplied by user.)
      [INFO] Skipped /workspace/target/generated-sources/openapi/README.md (Skipped by supportingFiles options supplied by user.)
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/ApiUtil.java
      [INFO] Skipped /workspace/target/generated-sources/openapi/.openapi-generator-ignore (Skipped by supportingFiles options supplied by user.)
      [INFO] writing file /workspace/target/generated-sources/openapi/.openapi-generator/VERSION
      [INFO] writing file /workspace/target/generated-sources/openapi/.openapi-generator/FILES
      ################################################################################
      # Thanks for using OpenAPI Generator.                                          #
      # Please consider donation to help us maintain this project ?                 #
      # https://opencollective.com/openapi_generator/donate                          #
      ################################################################################
      [INFO]
      [INFO] --- build-helper-maven-plugin:3.2.0:add-source (add-source) @ spring-petclinic-rest ---
      [INFO] Source directory: /workspace/target/generated-sources/openapi/src/main/java added.
      [INFO]
      [INFO] --- spring-boot-maven-plugin:2.6.2:build-info (default) @ spring-petclinic-rest ---
      [INFO]
      [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ spring-petclinic-rest ---
      [INFO] Using 'UTF-8' encoding to copy filtered resources.
      [INFO] Using 'UTF-8' encoding to copy filtered properties files.
      [INFO] Copying 4 resources
      [INFO] Copying 13 resources
      [INFO]
      [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ spring-petclinic-rest ---
      [INFO] Changes detected - recompiling the module!
      [INFO] Compiling 108 source files to /workspace/target/classes
      [INFO]
      [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ spring-petclinic-rest ---
      [INFO] Not copying test resources
      [INFO]
      [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ spring-petclinic-rest ---
      [INFO] Not compiling test sources
      [INFO]
      [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-petclinic-rest ---
      [INFO] Tests are skipped.
      [INFO]
      [INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ spring-petclinic-rest ---
      [INFO] Skipping JaCoCo execution due to missing execution data file.
      [INFO]
      [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ spring-petclinic-rest ---
      [INFO] Building jar: /workspace/target/spring-petclinic-rest-2.6.2.jar
      [INFO]
      [INFO] --- spring-boot-maven-plugin:2.6.2:repackage (repackage) @ spring-petclinic-rest ---
      [INFO] Replacing main artifact with repackaged archive
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  7.741 s
      [INFO] Finished at: 2022-01-16T10:43:45Z
      [INFO] ------------------------------------------------------------------------
  Removing source code

Paketo Executable JAR Buildpack 6.0.2
  https://github.com/paketo-buildpacks/executable-jar
  Class Path: Contributing to layer
    Writing env/CLASSPATH.delim
    Writing env/CLASSPATH.prepend
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
    task:           java org.springframework.boot.loader.JarLauncher (direct)
    web:            java org.springframework.boot.loader.JarLauncher (direct)

Paketo Spring Boot Buildpack 5.3.0
  https://github.com/paketo-buildpacks/spring-boot
  Creating slices from layers index
    dependencies
    spring-boot-loader
    snapshot-dependencies
    application
  Launch Helper: Reusing cached layer
  Spring Cloud Bindings 1.8.0: Reusing cached layer
  Web Application Type: Contributing to layer
    Servlet web application detected
    Writing env.launch/BPL_JVM_THREAD_COUNT.default
  4 application slices
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.version
===> EXPORT
Reusing layers from image 'harbor.registry.com/apps/spring-petclinic-backend@sha256:6cf16c4573c95f5e66e40740d9f6238377b3aaab38176e7986310c21bdd6094a'
Reusing layer 'paketo-buildpacks/ca-certificates:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:jre'
Reusing layer 'paketo-buildpacks/executable-jar:classpath'
Reusing layer 'paketo-buildpacks/spring-boot:helper'
Reusing layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Reusing layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding layer 'launch.sbom'
Reusing 4/5 app layer(s)
Adding 1/5 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving harbor.registry.com/apps/spring-petclinic-backend...
*** Images (sha256:d55635a51867aa722318fd405b288d865c42604ad758804bb20ef913e88e111e):
      harbor.registry.com/apps/spring-petclinic-backend
      harbor.registry.com/apps/spring-petclinic-backend:b2.20220116.104312
Reusing cache layer 'paketo-buildpacks/bellsoft-liberica:jdk'
Reusing cache layer 'paketo-buildpacks/syft:syft'
Adding cache layer 'paketo-buildpacks/maven:application'
Reusing cache layer 'paketo-buildpacks/maven:cache'
Reusing cache layer 'cache.sbom'
===> COMPLETION
Build successful

Let's run the image again:

> docker run -p 8080:8080 harbor.registry.com/apps/spring-petclinic-backend:b2.20220116.104312
Unable to find image 'harbor.registry.com/apps/spring-petclinic-backend:b2.20220116.104312' locally
b2.20220116.104312: Pulling from apps/spring-petclinic-backend
bf99a8b93828: Already exists
1da1131b1360: Already exists
48bde579d8cc: Already exists
1929d0623312: Already exists
1fea93ec33be: Already exists
fed2bcb93d83: Already exists
d06e3ac07eba: Already exists
a8090bf3b91d: Already exists
204c9b3ef04c: Already exists
95d9537f4e2e: Already exists
40cbb147917e: Already exists
b8e5cf599c18: Already exists
bd74081a3b16: Already exists
1eeff7ac260d: Pull complete
b218c24b76da: Pull complete
afd258a43f9e: Pull complete
89732bc75041: Pull complete
ee556fcb0298: Pull complete
40d17e1b116e: Pull complete
c3e95d93d7ac: Pull complete
b39b18afe74e: Pull complete
Digest: sha256:d55635a51867aa722318fd405b288d865c42604ad758804bb20ef913e88e111e
Status: Downloaded newer image for harbor.registry.com/apps/spring-petclinic-backend:b2.20220116.104312
Setting Active Processor Count to 4
WARNING: Unable to convert memory limit "max" from path "/sys/fs/cgroup/memory.max" as int: memory size "max" does not match pattern "^([\\d]+)([kmgtKMGT]?)$"
Calculating JVM memory based on 1279512K available memory
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx643509K -XX:MaxMetaspaceSize=124002K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1279512K, Thread Count: 250, Loaded Class Count: 19479, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 128 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx643509K -XX:MaxMetaspaceSize=124002K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.2)

INFO  PetClinicApplication - Starting PetClinicApplication v2.6.2 using Java 11.0.13 on 8c17a9943dd9 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
DEBUG PetClinicApplication - Running with Spring Boot v2.6.2, Spring v5.3.14
INFO  PetClinicApplication - The following profiles are active: hsqldb,spring-data-jpa
INFO  Version - HV000001: Hibernate Validator 6.2.0.Final
INFO  RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
INFO  RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 81 ms. Found 7 JPA repository interfaces.
INFO  TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
INFO  Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
INFO  StandardService - Starting service [Tomcat]
INFO  StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.56]
INFO  [/petclinic] - Initializing Spring embedded WebApplicationContext
INFO  ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 2294 ms
INFO  HikariDataSource - HikariPool-1 - Starting...
INFO  PoolBase - HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
INFO  HikariDataSource - HikariPool-1 - Start completed.
INFO  LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
INFO  Version - HHH000412: Hibernate ORM core version 5.6.3.Final
INFO  Version - HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
INFO  Dialect - HHH000400: Using dialect: org.hibernate.dialect.HSQLDialect
INFO  JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
INFO  LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default'
INFO  EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path '/actuator'
INFO  UserDetailsServiceAutoConfiguration -

Using generated security password: 73e3ef49-bc5c-4922-a168-570d4d8c048e

INFO  DefaultSecurityFilterChain - Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@62cbe18c, org.springframework.security.web.context.SecurityContextPersistenceFilter@490b8f1e, org.springframework.security.web.header.HeaderWriterFilter@dc6a186, org.springframework.security.web.authentication.logout.LogoutFilter@35ef158f, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@4eb43d2f, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@5c70eda4, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@377b7798, org.springframework.security.web.session.SessionManagementFilter@27b9d5b7, org.springframework.security.web.access.ExceptionTranslationFilter@71cdde0c, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@6d398b6c]
INFO  Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
INFO  TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path '/petclinic'
INFO  PetClinicApplication - Started PetClinicApplication in 7.167 seconds (JVM running for 7.718)

The result looks much better now:

Changing the defaults

If we'll take a closer look at each buildpack stage in the logs, we'll see that there are certain defaults that are being applied to our image. For example:

  Build Configuration:
    $BP_JVM_TYPE                 JRE             the JVM type - JDK or JRE
    $BP_JVM_VERSION              11              the Java version
  Build Configuration:
    $BP_MAVEN_BUILD_ARGUMENTS  -Dmaven.test.skip=true package  the arguments to pass to Maven

Each one of these parameters can be changed in order to customize our build. For example - I may want to use Java 17 since it's 2022. I may also opt to run all unit tests as part of my build. In that case, I could change the parameters as follows:

• BP_JVM_VERSION=17
• BP_MAVEN_BUILD_ARGUMENTS="-Dmaven.test.skip=false package" Let's update our build accordingly:

> kp image save spring-petclinic-backend \
    --tag harbor.registry.com/apps/spring-petclinic-backend \
    --git https://github.com/odedia/spring-petclinic-rest.git \
    --git-revision master \
    --env BP_JVM_VERSION=17 \
    --env BP_MAVEN_BUILD_ARGUMENTS="-Dmaven.test.skip=false package"

If we'll check the updated logs, we'll see that we now use Java 17 and that we are now running all unit tests (logs truncated due to many lines of unit testing...):

> kp build logs spring-petclinic-backend
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  + env:
  + - name: BP_JVM_VERSION
  +   value: "17"
  + - name: BP_MAVEN_BUILD_ARGUMENTS
  +   value: -Dmaven.test.skip=false package
  resources: {}
  source:
    git:
      revision: c31f00af9fae5ef63c323cfbf5f890f252d8669a
      url: https://github.com/odedia/spring-petclinic-rest.git
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/odedia/spring-petclinic-rest.git" @ "c31f00af9fae5ef63c323cfbf5f890f252d8669a"...
Successfully cloned "https://github.com/odedia/spring-petclinic-rest.git" @ "c31f00af9fae5ef63c323cfbf5f890f252d8669a" in path "/workspace"
===> ANALYZE
===> DETECT
8 of 35 buildpacks participating
paketo-buildpacks/ca-certificates   3.0.2
paketo-buildpacks/bellsoft-liberica 9.0.2
paketo-buildpacks/syft              1.3.1
paketo-buildpacks/maven             6.0.2
paketo-buildpacks/executable-jar    6.0.2
paketo-buildpacks/apache-tomcat     7.0.3
paketo-buildpacks/dist-zip          5.0.2
paketo-buildpacks/spring-boot       5.3.0
===> RESTORE
Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:helper" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:java-security-properties" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jre" from app image
Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jdk" from cache
Restoring metadata for "paketo-buildpacks/syft:syft" from cache
Restoring metadata for "paketo-buildpacks/maven:application" from cache
Restoring metadata for "paketo-buildpacks/maven:cache" from cache
Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
Restoring data for "paketo-buildpacks/bellsoft-liberica:jdk" from cache
Restoring data for "paketo-buildpacks/syft:syft" from cache
Restoring data for "paketo-buildpacks/maven:application" from cache
Restoring data for "paketo-buildpacks/maven:cache" from cache
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Reusing cached layer

Paketo BellSoft Liberica Buildpack 9.0.2
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_TYPE                 JRE             the JVM type - JDK or JRE
    $BP_JVM_VERSION              17              the Java version
  Launch Configuration:
    $BPL_DEBUG_ENABLED           false           enables Java remote debugging support
    $BPL_DEBUG_PORT              8000            configure the remote debugging port
    $BPL_DEBUG_SUSPEND           false           configure whether to suspend execution until a debugger has attached
    $BPL_HEAP_DUMP_PATH                          write heap dumps on error to this path
    $BPL_JAVA_NMT_ENABLED        true            enables Java Native Memory Tracking (NMT)
    $BPL_JAVA_NMT_LEVEL          summary         configure level of NMT, summary or detail
    $BPL_JFR_ARGS                                configure custom Java Flight Recording (JFR) arguments
    $BPL_JFR_ENABLED             false           enables Java Flight Recording (JFR)
    $BPL_JMX_ENABLED             false           enables Java Management Extensions (JMX)
    $BPL_JMX_PORT                5000            configure the JMX port
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags
  BellSoft Liberica JDK 17.0.1: Contributing to layer
    Reusing cached download from buildpack
    Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jdk
    Adding 128 container CA certificates to JVM truststore
    Writing env.build/JAVA_HOME.override
    Writing env.build/JDK_HOME.override
  BellSoft Liberica JRE 17.0.1: Contributing to layer
    Reusing cached download from buildpack
    Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
    Adding 128 container CA certificates to JVM truststore
    Writing env.launch/BPI_APPLICATION_PATH.default
    Writing env.launch/BPI_JVM_CACERTS.default
    Writing env.launch/BPI_JVM_CLASS_COUNT.default
    Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
    Writing env.launch/JAVA_HOME.default
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim
    Writing env.launch/MALLOC_ARENA_MAX.default
  Launch Helper: Reusing cached layer
  Java Security Properties: Reusing cached layer

Paketo Syft Buildpack 1.3.1
  https://github.com/paketo-buildpacks/syft

Paketo Maven Buildpack 6.0.2
  https://github.com/paketo-buildpacks/maven
  Build Configuration:
    $BP_MAVEN_BUILD_ARGUMENTS  -Dmaven.test.skip=false package  the arguments to pass to Maven
    $BP_MAVEN_BUILT_ARTIFACT   target/*.[ejw]ar                 the built application artifact explicitly.  Supersedes $BP_MAVEN_BUILT_MODULE
    $BP_MAVEN_BUILT_MODULE                                      the module to find application artifact in
    $BP_MAVEN_POM_FILE         pom.xml                          the location of the main pom.xml file, relative to the application root
    Creating cache directory /home/cnb/.m2
  Compiled Application: Contributing to layer
    Executing mvnw --batch-mode -Dmaven.test.skip=false package
      [INFO] Scanning for projects...
      [INFO]
      [INFO] ---------< org.springframework.samples:spring-petclinic-rest >----------
      [INFO] Building spring-petclinic-rest 2.6.2
      [INFO] --------------------------------[ jar ]---------------------------------
      [INFO]
      [INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (default) @ spring-petclinic-rest ---
      [INFO] argLine set to -javaagent:/home/cnb/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/workspace/target/jacoco.exec,excludes=**/org/springframework/samples/petclinic/rest/dto/**:**/org/springframework/samples/petclinic/rest/api/**
      [INFO]
      [INFO] --- openapi-generator-maven-plugin:5.2.1:generate (default) @ spring-petclinic-rest ---
      [INFO] Generating with dryRun=false
      [INFO] Output directory (/workspace/target/generated-sources/openapi) does not exist, or is inaccessible. No file (.openapi-generator-ignore) will be evaluated.
      [INFO] OpenAPI Generator: spring (server)
      [INFO] Generator 'spring' is considered stable.
      [INFO] ----------------------------------
      [WARNING] java8 option has been deprecated as it's set to true by default (JDK7 support has been deprecated)
      [INFO] Environment variable JAVA_POST_PROCESS_FILE not defined so the Java code may not be properly formatted. To define it, try 'export JAVA_POST_PROCESS_FILE="/usr/local/bin/clang-format -i"' (Linux/Mac)
      [INFO] NOTE: To enable file post-processing, 'enablePostProcessFile' must be set to `true` (--enable-post-process-file for CLI).
      [INFO] Invoker Package Name, originally not set, is now derived from api package name: org.springframework.samples.petclinic.rest
      [INFO] Processing operation failingRequest
      [INFO] Processing operation listOwners
      [INFO] Processing operation addOwner
      [INFO] Processing operation getOwner
      [INFO] Processing operation updateOwner
      [INFO] Processing operation addPet
      [INFO] Processing operation getPet
      [INFO] Processing operation updatePet
      [INFO] Processing operation addVisit
      [INFO] Processing operation listPetTypes
      [INFO] Processing operation listVets
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/OwnerAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/OwnerDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/OwnerFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetTypeAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetTypeDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/PetTypeFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/RestErrorDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/RoleDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/SpecialtyDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/UserDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/ValidationMessageDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VetAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VetDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VetFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VisitAllOfDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VisitDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/dto/VisitFieldsDto.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/OopsApi.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/OwnersApi.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/PettypesApi.java
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/VetsApi.java
      [INFO] Skipped /workspace/target/generated-sources/openapi/pom.xml (Skipped by supportingFiles options supplied by user.)
      [INFO] Skipped /workspace/target/generated-sources/openapi/README.md (Skipped by supportingFiles options supplied by user.)
      [INFO] writing file /workspace/target/generated-sources/openapi/src/main/java/org/springframework/samples/petclinic/rest/api/ApiUtil.java
      [INFO] Skipped /workspace/target/generated-sources/openapi/.openapi-generator-ignore (Skipped by supportingFiles options supplied by user.)
      [INFO] writing file /workspace/target/generated-sources/openapi/.openapi-generator/VERSION
      [INFO] writing file /workspace/target/generated-sources/openapi/.openapi-generator/FILES
      ################################################################################
      # Thanks for using OpenAPI Generator.                                          #
      # Please consider donation to help us maintain this project ?                 #
      # https://opencollective.com/openapi_generator/donate                          #
      ################################################################################
      [INFO]
      [INFO] --- build-helper-maven-plugin:3.2.0:add-source (add-source) @ spring-petclinic-rest ---
      [INFO] Source directory: /workspace/target/generated-sources/openapi/src/main/java added.
      [INFO]
      [INFO] --- spring-boot-maven-plugin:2.6.2:build-info (default) @ spring-petclinic-rest ---
      [INFO]
      [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ spring-petclinic-rest ---
      [INFO] Using 'UTF-8' encoding to copy filtered resources.
      [INFO] Using 'UTF-8' encoding to copy filtered properties files.
      [INFO] Copying 4 resources
      [INFO] Copying 13 resources
      [INFO]
      [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ spring-petclinic-rest ---
      [INFO] Changes detected - recompiling the module!
      [INFO] Compiling 108 source files to /workspace/target/classes
      [INFO]
      [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ spring-petclinic-rest ---
      [INFO] Using 'UTF-8' encoding to copy filtered resources.
      [INFO] Using 'UTF-8' encoding to copy filtered properties files.
      [INFO] Copying 1 resource
      [INFO]
      [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ spring-petclinic-rest ---
      [INFO] Changes detected - recompiling the module!
      [INFO] Compiling 18 source files to /workspace/target/test-classes
      [INFO]
      [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-petclinic-rest ---
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/2.22.2/surefire-junit-platform-2.22.2.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/2.22.2/surefire-junit-platform-2.22.2.pom (7.0 kB at 28 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.22.2/surefire-providers-2.22.2.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.22.2/surefire-providers-2.22.2.pom (2.5 kB at 113 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.3.1/junit-platform-launcher-1.3.1.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.3.1/junit-platform-launcher-1.3.1.pom (2.2 kB at 222 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.0.0/apiguardian-api-1.0.0.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.0.0/apiguardian-api-1.0.0.pom (1.2 kB at 146 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.3.1/junit-platform-engine-1.3.1.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.3.1/junit-platform-engine-1.3.1.pom (2.4 kB at 126 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.3.1/junit-platform-commons-1.3.1.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.3.1/junit-platform-commons-1.3.1.pom (2.0 kB at 202 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.1.1/opentest4j-1.1.1.pom
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.1.1/opentest4j-1.1.1.pom (1.7 kB at 129 kB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/2.22.2/surefire-junit-platform-2.22.2.jar
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.3.1/junit-platform-launcher-1.3.1.jar
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.3.1/junit-platform-commons-1.3.1.jar
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.0.0/apiguardian-api-1.0.0.jar
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.3.1/junit-platform-engine-1.3.1.jar
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apiguardian/apiguardian-api/1.0.0/apiguardian-api-1.0.0.jar (2.2 kB at 37 kB/s)
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit-platform/2.22.2/surefire-junit-platform-2.22.2.jar (66 kB at 1.0 MB/s)
      [INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.1.1/opentest4j-1.1.1.jar
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/opentest4j/opentest4j/1.1.1/opentest4j-1.1.1.jar (7.1 kB at 647 kB/s)
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-launcher/1.3.1/junit-platform-launcher-1.3.1.jar (95 kB at 1.1 MB/s)
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-commons/1.3.1/junit-platform-commons-1.3.1.jar (78 kB at 771 kB/s)
      [INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/junit/platform/junit-platform-engine/1.3.1/junit-platform-engine-1.3.1.jar (135 kB at 1.3 MB/s)
      [INFO]
      [INFO] -------------------------------------------------------
      [INFO]  T E S T S
      [INFO] -------------------------------------------------------
      [INFO] Running org.springframework.samples.petclinic.rest.controller.OwnerRestControllerTests
      INFO  AbstractContextLoader - Could not detect default resource locations for test class [org.springframework.samples.petclinic.rest.controller.OwnerRestControllerTests]: no resource found for suffixes {-context.xml, Context.groovy}.
      INFO  SpringBootTestContextBootstrapper - Found @SpringBootConfiguration org.springframework.samples.petclinic.PetClinicApplication for test class org.springframework.samples.petclinic.rest.controller.OwnerRestControllerTests
      INFO  SpringBootTestContextBootstrapper - Loaded default TestExecutionListener class names from location [META-INF/spring.factories]: [org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener, org.springframework.boot.test.autoconfigure.webservices.client.MockWebServiceServerTestExecutionListener, org.springframework.test.context.web.ServletTestExecutionListener, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener, org.springframework.test.context.event.ApplicationEventsTestExecutionListener, org.springframework.test.context.support.DependencyInjectionTestExecutionListener, org.springframework.test.context.support.DirtiesContextTestExecutionListener, org.springframework.test.context.transaction.TransactionalTestExecutionListener, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener, org.springframework.test.context.event.EventPublishingTestExecutionListener, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener, org.springframework.security.test.context.support.ReactorContextTestExecutionListener]
      INFO  SpringBootTestContextBootstrapper - Using TestExecutionListeners: [org.springframework.test.context.web.ServletTestExecutionListener@ea52184, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener@3e7b65d7, org.springframework.test.context.event.ApplicationEventsTestExecutionListener@3c854752, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener@3ddeaa5f, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener@1a500561, org.springframework.test.context.support.DirtiesContextTestExecutionListener@7f608e21, org.springframework.test.context.transaction.TransactionalTestExecutionListener@6949e948, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener@210d2a6c, org.springframework.test.context.event.EventPublishingTestExecutionListener@3bec5821, org.springframework.security.test.context.support.WithSecurityContextTestExecutionListener@4086d8fb, org.springframework.security.test.context.support.ReactorContextTestExecutionListener@66236a0a, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener@2e3572e8, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener@77c10a5f, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener@176555c, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener@7c781c42, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener@795f8317, org.springframework.boot.test.autoconfigure.webservices.client.MockWebServiceServerTestExecutionListener@633fd91]

        .   ____          _            __ _ _
       /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
      ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
       \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
        '  |____| .__|_| |_|_| |_\__, | / / / /
       =========|_|==============|___/=/_/_/_/
       :: Spring Boot ::                (v2.6.2)

      INFO  OwnerRestControllerTests - Starting OwnerRestControllerTests using Java 17.0.1 on spring-petclinic-backend-build-3-build-pod with PID 82 (started by cnb in /workspace)
      DEBUG OwnerRestControllerTests - Running with Spring Boot v2.6.2, Spring v5.3.14
      INFO  OwnerRestControllerTests - The following profiles are active: hsqldb,spring-data-jpa
      INFO  RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
      INFO  RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 108 ms. Found 7 JPA repository interfaces.
      INFO  HikariDataSource - HikariPool-1 - Starting...
      INFO  PoolBase - HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
      INFO  HikariDataSource - HikariPool-1 - Start completed.
      INFO  LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
      INFO  Version - HHH000412: Hibernate ORM core version 5.6.3.Final
      INFO  Version - HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
      INFO  Dialect - HHH000400: Using dialect: org.hibernate.dialect.HSQLDialect
      INFO  Version - HV000001: Hibernate Validator 6.2.0.Final
      INFO  JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
      INFO  LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default'
      INFO  EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path '/actuator'
      INFO  DefaultSecurityFilterChain - Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@4195e7fb, org.springframework.security.web.context.SecurityContextPersistenceFilter@3999f490, org.springframework.security.web.header.HeaderWriterFilter@152a8948, org.springframework.security.web.authentication.logout.LogoutFilter@20df4edc, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@191323f5, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@47ac1aa0, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@4fe30669, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@187ed1ef, org.springframework.security.web.session.SessionManagementFilter@7472bba0, org.springframework.security.web.access.ExceptionTranslationFilter@2104e947, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@7a4f9db7]
      INFO  OwnerRestControllerTests - Started OwnerRestControllerTests in 10.988 seconds (JVM running for 13.299)
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      --> newPetAsJSON={"name":"Rosy","birthDate":"2022-01-16","type":{"name":"dog","id":2},"id":999,"ownerId":null,"visits":[]}
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 1 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms

      MockHttpServletRequest:
            HTTP Method = POST
            Request URI = /api/owners/1/pets/
             Parameters = {}
                Headers = [Content-Type:"application/json", Accept:"application/json", Content-Length:"104"]
                   Body = <no character encoding set>
          Session Attrs = {}

      Handler:
                   Type = org.springframework.samples.petclinic.rest.controller.OwnerRestController
                 Method = org.springframework.samples.petclinic.rest.controller.OwnerRestController#addPet(Integer, PetFieldsDto)

      Async:
          Async started = false
           Async result = null

      Resolved Exception:
                   Type = org.springframework.web.bind.MethodArgumentNotValidException

      ModelAndView:
              View name = null
                   View = null
                  Model = null

      FlashMap:
             Attributes = null

      MockHttpServletResponse:
                 Status = 400
          Error message = null
                Headers = [Vary:"Origin", "Access-Control-Request-Method", "Access-Control-Request-Headers", Content-Type:"application/json", Content-Length:"737"]
           Content type = application/json
                   Body = {"className":"org.springframework.web.bind.MethodArgumentNotValidException","exMessage":"Validation failed for argument [1] in public org.springframework.http.ResponseEntity<org.springframework.samples.petclinic.rest.dto.PetDto> org.springframework.samples.petclinic.rest.controller.OwnerRestController.addPet(java.lang.Integer,org.springframework.samples.petclinic.rest.dto.PetFieldsDto): [Field error in object 'petFieldsDto' on field 'name': rejected value [null]; codes [NotNull.petFieldsDto.name,NotNull.name,NotNull.java.lang.String,NotNull]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [petFieldsDto.name,name]; arguments []; default message [name]]; default message [must not be null]] "}
          Forwarded URL = null
         Redirected URL = null
                Cookies = []
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 5 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      newVisitAsJSON {"id":999,"date":"2022-01-16","description":"rabies shot","pet":null}
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 1 ms
      INFO  MockServletContext - Initializing Spring TestDispatcherServlet ''
      INFO  TestDispatcherServlet - Initializing Servlet ''
      INFO  TestDispatcherServlet - Completed initialization in 0 ms
      [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.43 s - in org.springframework.samples.petclinic.rest.controller.OwnerRestControllerTests
      [INFO] Running org.springframework.samples.petclinic.rest.controller.SpecialtyRestControllerTests
      INFO  AbstractContextLoader - Could not detect default resource locations for test class [org.springframework.samples.petclinic.rest.controller.SpecialtyRestControllerTests]: no resource found for suffixes {-context.xml, Context.groovy}.
      INFO  SpringBootTestContextBootstrapper - Found @SpringBootConfiguration org.springframework.samples.petclinic.PetClinicApplication for test class org.springframework.samples.petclinic.rest.controller.SpecialtyRestControllerTests
...
...
      INFO  DefaultSecurityFilterChain - Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@4af026c6, org.springframework.security.web.context.SecurityContextPersistenceFilter@2ce17213, org.springframework.security.web.header.HeaderWriterFilter@29d76031, org.springframework.security.web.authentication.logout.LogoutFilter@2a97b5a2, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@72b927c6, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@5295b114, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@40d1859f, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@268433c9, org.springframework.security.web.session.SessionManagementFilter@64653709, org.springframework.security.web.access.ExceptionTranslationFilter@1dac64c7, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@a53b06d]
      INFO  SpringConfigTests - Started SpringConfigTests in 1.855 seconds (JVM running for 29.56)
      [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.845 s - in org.springframework.samples.petclinic.SpringConfigTests
      INFO  LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
      INFO  HikariDataSource - HikariPool-1 - Shutdown initiated...
      INFO  HikariDataSource - HikariPool-1 - Shutdown completed.
      INFO  LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
      INFO  HikariDataSource - HikariPool-2 - Shutdown initiated...
      INFO  HikariDataSource - HikariPool-2 - Shutdown completed.
      INFO  LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
      INFO  HikariDataSource - HikariPool-3 - Shutdown initiated...
      INFO  HikariDataSource - HikariPool-3 - Shutdown completed.
      INFO  LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
      INFO  HikariDataSource - HikariPool-4 - Shutdown initiated...
      INFO  HikariDataSource - HikariPool-4 - Shutdown completed.
      INFO  LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
      INFO  HikariDataSource - HikariPool-5 - Shutdown initiated...
      INFO  HikariDataSource - HikariPool-5 - Shutdown completed.
      INFO  LocalContainerEntityManagerFactoryBean - Closing JPA EntityManagerFactory for persistence unit 'default'
      INFO  HikariDataSource - HikariPool-6 - Shutdown initiated...
      INFO  HikariDataSource - HikariPool-6 - Shutdown completed.
      [INFO]
      [INFO] Results:
      [INFO]
      [INFO] Tests run: 172, Failures: 0, Errors: 0, Skipped: 0
      [INFO]
      [INFO]
      [INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ spring-petclinic-rest ---
      [INFO] Loading execution data file /workspace/target/jacoco.exec
      [INFO] Analyzed bundle 'spring-petclinic-rest' with 65 classes
      [INFO]
      [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ spring-petclinic-rest ---
      [INFO] Building jar: /workspace/target/spring-petclinic-rest-2.6.2.jar
      [INFO]
      [INFO] --- spring-boot-maven-plugin:2.6.2:repackage (repackage) @ spring-petclinic-rest ---
      [INFO] Replacing main artifact with repackaged archive
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  39.657 s
      [INFO] Finished at: 2022-01-16T10:58:22Z
      [INFO] ------------------------------------------------------------------------
  Removing source code

Paketo Executable JAR Buildpack 6.0.2
  https://github.com/paketo-buildpacks/executable-jar
  Class Path: Contributing to layer
    Writing env/CLASSPATH.delim
    Writing env/CLASSPATH.prepend
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
    task:           java org.springframework.boot.loader.JarLauncher (direct)
    web:            java org.springframework.boot.loader.JarLauncher (direct)

Paketo Spring Boot Buildpack 5.3.0
  https://github.com/paketo-buildpacks/spring-boot
  Creating slices from layers index
    dependencies
    spring-boot-loader
    snapshot-dependencies
    application
  Launch Helper: Reusing cached layer
  Spring Cloud Bindings 1.8.0: Reusing cached layer
  Web Application Type: Contributing to layer
    Servlet web application detected
    Writing env.launch/BPL_JVM_THREAD_COUNT.default
  4 application slices
  Image labels:
    org.opencontainers.image.title
    org.opencontainers.image.version
    org.springframework.boot.version
===> EXPORT
Reusing layers from image 'harbor.registry.com/apps/spring-petclinic-backend@sha256:d55635a51867aa722318fd405b288d865c42604ad758804bb20ef913e88e111e'
Reusing layer 'paketo-buildpacks/ca-certificates:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:helper'
Reusing layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
Adding layer 'paketo-buildpacks/bellsoft-liberica:jre'
Reusing layer 'paketo-buildpacks/executable-jar:classpath'
Reusing layer 'paketo-buildpacks/spring-boot:helper'
Reusing layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Reusing layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding layer 'launch.sbom'
Reusing 4/5 app layer(s)
Adding 1/5 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.opencontainers.image.title'
Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving harbor.registry.com/apps/spring-petclinic-backend...
*** Images (sha256:8096781a5d7236c037ef39a128a99eae397ba733d80600c515df09b93228067b):
      harbor.registry.com/apps/spring-petclinic-backend
      harbor.registry.com/apps/spring-petclinic-backend:b3.20220116.105703
Adding cache layer 'paketo-buildpacks/bellsoft-liberica:jdk'
Reusing cache layer 'paketo-buildpacks/syft:syft'
Adding cache layer 'paketo-buildpacks/maven:application'
Adding cache layer 'paketo-buildpacks/maven:cache'
Adding cache layer 'cache.sbom'
===> COMPLETION
Build successful

Let's run the updated image:

> docker run -p 8080:8080 harbor.registry.com/apps/spring-petclinic-backend:b3.20220116.105703
Setting Active Processor Count to 4
WARNING: Unable to convert memory limit "max" from path "/sys/fs/cgroup/memory.max" as int: memory size "max" does not match pattern "^([\\d]+)([kmgtKMGT]?)$"
Calculating JVM memory based on 1283900K available memory
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx655838K -XX:MaxMetaspaceSize=116061K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1283900K, Thread Count: 250, Loaded Class Count: 18077, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 128 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx655838K -XX:MaxMetaspaceSize=116061K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.2)

INFO  Version - HV000001: Hibernate Validator 6.2.0.Final
INFO  PetClinicApplication - Starting PetClinicApplication v2.6.2 using Java 17.0.1 on de3b2736ea60 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
DEBUG PetClinicApplication - Running with Spring Boot v2.6.2, Spring v5.3.14
INFO  PetClinicApplication - The following profiles are active: hsqldb,spring-data-jpa
INFO  RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
INFO  RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 80 ms. Found 7 JPA repository interfaces.
INFO  TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
INFO  Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
INFO  StandardService - Starting service [Tomcat]
INFO  StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.56]
INFO  [/petclinic] - Initializing Spring embedded WebApplicationContext
INFO  ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 2020 ms
INFO  HikariDataSource - HikariPool-1 - Starting...
INFO  PoolBase - HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
INFO  HikariDataSource - HikariPool-1 - Start completed.
INFO  LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
INFO  Version - HHH000412: Hibernate ORM core version 5.6.3.Final
INFO  Version - HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
INFO  Dialect - HHH000400: Using dialect: org.hibernate.dialect.HSQLDialect
INFO  JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
INFO  LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default'
INFO  EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path '/actuator'
INFO  UserDetailsServiceAutoConfiguration -

Using generated security password: 86095fd7-2e96-4455-bf12-c85a821c51b0

INFO  DefaultSecurityFilterChain - Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@18206382, org.springframework.security.web.context.SecurityContextPersistenceFilter@6030eff1, org.springframework.security.web.header.HeaderWriterFilter@cec1ce2, org.springframework.security.web.authentication.logout.LogoutFilter@311eb725, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@5a02fca5, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@309ac89e, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@3e8c9981, org.springframework.security.web.session.SessionManagementFilter@730611a0, org.springframework.security.web.access.ExceptionTranslationFilter@38320819, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@38816a6c]
INFO  Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
INFO  TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path '/petclinic'
INFO  PetClinicApplication - Started PetClinicApplication in 6.352 seconds (JVM running for 6.824)

We can see from the logs that the JRE is now version 17, and also that our startup time has improved from 7.5 seconds to just under 6 seconds. The application still works as expected:

Some more tips and tricks

• If you have multiple microservices in a single git repo, use the --sub-path parameter to point Build Service to the relevant sub-path.
• The buildpack will honor any maven repo setting you may have in your source code. So, if you are running in an airgapped environment and target a local Nexus or Jfrog Artifactory repo, make sure your maven pom.xml or gradle build.settings are set with the correct local repo.
• If you don't want Build Service to kick off a new image automatically on each git commit, set the --git-revision parameter to the current git commit id and not the desired git branch. Then, as part of your pipeline, you can update the commit ID to kick off a new build. Here is a snippet you can include in your CI pipeline:

export COMMIT_ID=`git rev-parse HEAD`
echo Commit ID is: $COMMIT_ID
kp image save spring-petclinic-backend \
    --tag harbor.registry.com/apps/spring-petclinic-backend \
    --git https://github.com/odedia/spring-petclinic-rest.git \
    --git-revision $COMMIT_ID \
    --env BP_JVM_VERSION=17 \
    --env BP_MAVEN_BUILD_ARGUMENTS="-Dmaven.test.skip=false package"

• If you want to extract the SHA256 of the last build image, you can use the below kubectl command:

  export LATEST_IMAGE_SHA=`kubectl get images.kpack.io spring-petclinic-backend -o json | jq .status.latestImage`
  echo Latest image sha256 is: $LATEST_IMAGE_SHA
  

.NET Core eShopWeb

.NET Core is the second most popular language I see in the field. It had its share of changes and turmoils with the move from .NET Framework to .NET Core 3, and now "Just .NET 5" and .NET 6. Let's try to build the popular eShopWeb .NET Core sample application.

Git repo

The git repository is available at https://github.com/dotnet-architecture/eShopOnWeb.git

Build command

Let's build the image:

> kp image save dotnet-eshopweb \
> --tag harbor.registry.com/apps/dotnet-eshopweb \
> --git https://github.com/dotnet-architecture/eShopOnWeb.git \
> --git-revision main
Creating Image Resource...
Image Resource "dotnet-eshopweb" created

Build logs output

Let's check the logs:

> kp build logs dotnet-eshopweb
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  - source: {}
  + source:
  +   git:
  +     revision: 896fae150fbcec5de8a1fb923eab5dfafe09422a
  +     url: https://github.com/dotnet-architecture/eShopOnWeb.git
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a"...
Successfully cloned "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a" in path "/workspace"
===> ANALYZE
Previous image with name "harbor.registry.com/apps/dotnet-eshopweb" not found
===> DETECT
ERROR: No buildpack groups passed detection.
ERROR: Please check that you are running against the correct path.
ERROR: failed to detect: no buildpacks participating

Well that didn't work... What happened here?

It seems that, like many .NET Core projects, the source code is nested inside our main repo. Let's try to build the ApplicationCore module by pointing at the sub path:

> kp image save dotnet-eshopweb \
--tag harbor.registry.com/apps/dotnet-eshopweb \
--git https://github.com/dotnet-architecture/eShopOnWeb.git \
--git-revision main \
--sub-path ./src/Web

Let's check the updated logs:

> kp build logs dotnet-eshopweb
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  - source: {}
  + source:
  +   git:
  +     revision: 896fae150fbcec5de8a1fb923eab5dfafe09422a
  +     url: https://github.com/dotnet-architecture/eShopOnWeb.git
  +   subPath: ./src/Web
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a"...
Successfully cloned "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a" in path "/workspace"
===> ANALYZE
Previous image with name "harbor.registry.com/apps/dotnet-eshopweb" not found
===> DETECT
7 of 13 buildpacks participating
paketo-buildpacks/ca-certificates    3.0.2
tanzu-buildpacks/dotnet-core-runtime 0.3.1
tanzu-buildpacks/dotnet-core-aspnet  0.3.2
tanzu-buildpacks/dotnet-core-sdk     0.3.1
tanzu-buildpacks/icu                 0.0.13
tanzu-buildpacks/dotnet-publish      0.5.1
tanzu-buildpacks/dotnet-execute      0.4.1
===> RESTORE
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Tanzu .NET Core Runtime Buildpack 0.3.1
  Resolving Dotnet Core Runtime version
    Candidate version sources (in priority order):
      Web.csproj -> "6.0.0"
      <unknown>  -> ""

    Selected dotnet-runtime version (using Web.csproj): 6.0.0

  Executing build process
    Installing Dotnet Core Runtime 6.0.0
      Completed in 4.228s

  Configuring environment for build and launch
    DOTNET_ROOT -> "/workspace/.dotnet_root"

  Configuring environment for build
    RUNTIME_VERSION -> "6.0.0"

Tanzu ASP.NET Core Buildpack 0.3.2
  Resolving Dotnet Core ASPNet version
    Candidate version sources (in priority order):
      RUNTIME_VERSION -> "6.0.0"
      Web.csproj      -> "6.0.0"
      <unknown>       -> ""

    Selected dotnet-aspnetcore version (using RUNTIME_VERSION): 6.0.0

  Executing build process
    Installing Dotnet Core ASPNet 6.0.0
      Completed in 1.292s

  Configuring environment
    DOTNET_ROOT -> "/workspace/.dotnet_root"

Tanzu .NET Core SDK Buildpack 0.3.1
  Resolving .NET Core SDK version
    Candidate version sources (in priority order):
      RUNTIME_VERSION -> "6.0.100"
      <unknown>       -> "*"
      Web.csproj      -> "6.0.*"

    Selected .NET Core SDK version (using RUNTIME_VERSION): 6.0.100

  Executing build process
    Installing .NET Core SDK 6.0.100
      Completed in 16.038s

  Configuring environment
    DOTNET_ROOT -> "/workspace/.dotnet_root"
    PATH        -> "/workspace/.dotnet_root:$PATH"

Tanzu ICU Buildpack 0.0.13
  Executing build process
    Installing ICU
      Completed in 480ms

Tanzu .Net Publish Buildpack 0.5.1
  Executing build process
    Running 'dotnet publish /workspace --configuration Release --runtime ubuntu.18.04-x64 --self-contained false --output /tmp/dotnet-publish-output2505033169'
      Failed after 20.780801107s
failed to execute 'dotnet publish': exit status 1

        Welcome to .NET 6.0!
        ---------------------
        SDK Version: 6.0.100

        Telemetry
        ---------
        The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

        Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

        ----------------
        Installed an ASP.NET Core HTTPS development certificate.
        To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
        Learn about HTTPS: https://aka.ms/dotnet-https
        ----------------
        Write your first app: https://aka.ms/dotnet-hello-world
        Find out what's new: https://aka.ms/dotnet-whats-new
        Explore documentation: https://aka.ms/dotnet-docs
        Report issues and find source on GitHub: https://github.com/dotnet/core
        Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
        --------------------------------------------------------------------------------------
        Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
        Copyright (C) Microsoft Corporation. All rights reserved.

          Determining projects to restore...
          Skipping project "/ApplicationCore/ApplicationCore.csproj" because it was not found.
          Skipping project "/BlazorAdmin/BlazorAdmin.csproj" because it was not found.
          Skipping project "/BlazorShared/BlazorShared.csproj" because it was not found.
          Skipping project "/Infrastructure/Infrastructure.csproj" because it was not found.
          Skipping project "/ApplicationCore/ApplicationCore.csproj" because it was not found.
          Skipping project "/BlazorAdmin/BlazorAdmin.csproj" because it was not found.
          Skipping project "/BlazorShared/BlazorShared.csproj" because it was not found.
          Skipping project "/Infrastructure/Infrastructure.csproj" because it was not found.
          Restored /workspace/Web.csproj (in 14.4 sec).

          Restore operation started...

          6 libraries restored in 0.56 seconds

        /layers/tanzu-buildpacks_dotnet-core-sdk/dotnet-core-sdk/sdk/6.0.100/Microsoft.Common.CurrentVersion.targets(2068,5): warning : The referenced project '../ApplicationCore/ApplicationCore.csproj' does not exist. [/workspace/Web.csproj]
        /layers/tanzu-buildpacks_dotnet-core-sdk/dotnet-core-sdk/sdk/6.0.100/Microsoft.Common.CurrentVersion.targets(2068,5): warning : The referenced project '../BlazorAdmin/BlazorAdmin.csproj' does not exist. [/workspace/Web.csproj]
        /layers/tanzu-buildpacks_dotnet-core-sdk/dotnet-core-sdk/sdk/6.0.100/Microsoft.Common.CurrentVersion.targets(2068,5): warning : The referenced project '../BlazorShared/BlazorShared.csproj' does not exist. [/workspace/Web.csproj]
        /layers/tanzu-buildpacks_dotnet-core-sdk/dotnet-core-sdk/sdk/6.0.100/Microsoft.Common.CurrentVersion.targets(2068,5): warning : The referenced project '../Infrastructure/Infrastructure.csproj' does not exist. [/workspace/Web.csproj]

          Bundler: Begin processing bundleconfig.json
          Bundler: Done processing bundleconfig.json
        /workspace/Areas/Identity/Pages/Account/ConfirmEmail.cshtml.cs(9,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Login.cshtml.cs(12,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Login.cshtml.cs(13,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Logout.cshtml.cs(10,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(11,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Configuration/ConfigureCoreServices.cs(1,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Configuration/ConfigureCoreServices.cs(2,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Configuration/ConfigureCoreServices.cs(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Configuration/ConfigureCoreServices.cs(4,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Configuration/ConfigureCoreServices.cs(5,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Configuration/ConfigureCoreServices.cs(6,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(10,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(11,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/UserController.cs(5,7): error CS0246: The type or namespace name 'BlazorShared' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/UserController.cs(8,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Extensions/EmailSenderExtensions.cs(3,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(6,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(7,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(8,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(5,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(6,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(7,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/HealthChecks/ApiHealthCheck.cs(4,7): error CS0246: The type or namespace name 'BlazorShared' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Interfaces/IBasketViewModelService.cs(1,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Admin/EditCatalogItem.cshtml.cs(5,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Admin/Index.cshtml.cs(4,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(11,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(12,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(13,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(14,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Index.cshtml.cs(8,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Shared/Components/BasketComponent/Basket.cs(5,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(3,7): error CS0246: The type or namespace name 'BlazorAdmin' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(4,7): error CS0246: The type or namespace name 'BlazorAdmin' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(5,7): error CS0246: The type or namespace name 'Blazored' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(6,7): error CS0246: The type or namespace name 'BlazorShared' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(13,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(14,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Program.cs(15,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(4,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(5,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(6,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(7,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogItemViewModelService.cs(2,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogItemViewModelService.cs(3,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(6,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(7,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(8,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/ViewModels/OrderViewModel.cs(3,26): error CS0234: The type or namespace name 'ApplicationCore' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/_ViewImports.cshtml(3,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/ConfirmEmail.cshtml.cs(16,34): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/ConfirmEmail.cshtml.cs(18,42): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(528,55): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(22,34): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(23,36): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(24,22): error CS0246: The type or namespace name 'IEmailSender' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(25,22): error CS0246: The type or namespace name 'IAppLogger<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(32,19): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(33,21): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(34,7): error CS0246: The type or namespace name 'IEmailSender' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/ManageController.cs(35,7): error CS0246: The type or namespace name 'IAppLogger<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(15,22): error CS0246: The type or namespace name 'IReadRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(15,38): error CS0246: The type or namespace name 'Order' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(17,31): error CS0246: The type or namespace name 'IReadRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/MyOrders/GetMyOrdersHandler.cs(17,47): error CS0246: The type or namespace name 'Order' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(14,22): error CS0246: The type or namespace name 'IReadRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(14,38): error CS0246: The type or namespace name 'Order' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Login.cshtml.cs(21,36): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/UserController.cs(29,24): error CS0246: The type or namespace name 'UserInfo' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(16,35): error CS0246: The type or namespace name 'IReadRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Features/OrderDetails/GetOrderDetailsHandler.cs(16,51): error CS0246: The type or namespace name 'Order' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/UserController.cs(16,22): error CS0246: The type or namespace name 'ITokenClaimsService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Login.cshtml.cs(23,22): error CS0246: The type or namespace name 'IBasketService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Controllers/UserController.cs(18,27): error CS0246: The type or namespace name 'ITokenClaimsService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/HealthChecks/ApiHealthCheck.cs(12,22): error CS0246: The type or namespace name 'BaseUrlConfiguration' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/HealthChecks/ApiHealthCheck.cs(14,36): error CS0246: The type or namespace name 'BaseUrlConfiguration' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Extensions/EmailSenderExtensions.cs(9,56): error CS0246: The type or namespace name 'IEmailSender' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Login.cshtml.cs(25,37): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Login.cshtml.cs(25,97): error CS0246: The type or namespace name 'IBasketService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(56,86): error CS0246: The type or namespace name 'BasketItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(80,44): error CS0246: The type or namespace name 'Basket' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Interfaces/IBasketViewModelService.cs(10,31): error CS0246: The type or namespace name 'Basket' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Logout.cshtml.cs(20,36): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(15,22): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(15,34): error CS0246: The type or namespace name 'Basket' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Logout.cshtml.cs(24,38): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Logout.cshtml.cs(35,64): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(16,22): error CS0246: The type or namespace name 'IUriComposer' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Admin/Index.cshtml(2,7): error CS0246: The type or namespace name 'BlazorAdmin' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Admin/EditCatalogItem.cshtml.cs(11,20): error CS0103: The name 'BlazorShared' does not exist in the current context [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(17,22): error CS0246: The type or namespace name 'IBasketQueryService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Admin/Index.cshtml.cs(12,20): error CS0103: The name 'BlazorShared' does not exist in the current context [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(19,36): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(18,22): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(18,34): error CS0246: The type or namespace name 'CatalogItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(20,34): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(25,21): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(26,23): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(60,42): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web.csproj]
        /workspace/Areas/Identity/Pages/Account/Register.cshtml.cs(65,69): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(20,35): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(20,47): error CS0246: The type or namespace name 'Basket' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(21,9): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(21,21): error CS0246: The type or namespace name 'CatalogItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(22,9): error CS0246: The type or namespace name 'IUriComposer' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/BasketViewModelService.cs(23,9): error CS0246: The type or namespace name 'IBasketQueryService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(22,22): error CS0246: The type or namespace name 'IBasketService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(23,36): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(24,22): error CS0246: The type or namespace name 'IOrderService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogItemViewModelService.cs(11,22): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogItemViewModelService.cs(11,34): error CS0246: The type or namespace name 'CatalogItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(27,22): error CS0246: The type or namespace name 'IAppLogger<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogItemViewModelService.cs(13,40): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogItemViewModelService.cs(13,52): error CS0246: The type or namespace name 'CatalogItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(21,22): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(21,34): error CS0246: The type or namespace name 'CatalogItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(29,26): error CS0246: The type or namespace name 'IBasketService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(31,23): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(32,9): error CS0246: The type or namespace name 'IOrderService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Checkout.cshtml.cs(33,9): error CS0246: The type or namespace name 'IAppLogger<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(22,22): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(22,34): error CS0246: The type or namespace name 'CatalogBrand' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Index.cshtml.cs(16,22): error CS0246: The type or namespace name 'IBasketService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Basket/Index.cshtml.cs(19,23): error CS0246: The type or namespace name 'IBasketService' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(23,22): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(23,34): error CS0246: The type or namespace name 'CatalogType' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(24,22): error CS0246: The type or namespace name 'IUriComposer' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(28,9): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(28,21): error CS0246: The type or namespace name 'CatalogItem' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(29,9): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(29,21): error CS0246: The type or namespace name 'CatalogBrand' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(30,9): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(30,21): error CS0246: The type or namespace name 'CatalogType' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Services/CatalogViewModelService.cs(31,9): error CS0246: The type or namespace name 'IUriComposer' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Shared/Components/BasketComponent/Basket.cs(14,36): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Pages/Shared/Components/BasketComponent/Basket.cs(17,35): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Microsoft.NET.Sdk.Razor.SourceGenerators/Microsoft.NET.Sdk.Razor.SourceGenerators.RazorSourceGenerator/Views_Manage__ManageNav_cshtml.g.cs(236,30): error CS0246: The type or namespace name 'ApplicationUser' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/Views/_ViewImports.cshtml(7,26): error CS0234: The type or namespace name 'Infrastructure' does not exist in the namespace 'Microsoft.eShopWeb' (are you missing an assembly reference?) [/workspace/Web.csproj]
        /workspace/SlugifyParameterTransformer.cs(8,19): warning CS8767: Nullability of reference types in type of parameter 'value' of 'string SlugifyParameterTransformer.TransformOutbound(object value)' doesn't match implicitly implemented member 'string? IOutboundParameterTransformer.TransformOutbound(object? value)' (possibly because of nullability attributes). [/workspace/Web.csproj]
        /workspace/ViewModels/OrderViewModel.cs(15,12): error CS0246: The type or namespace name 'Address' could not be found (are you missing a using directive or an assembly reference?) [/workspace/Web.csproj]
ERROR: failed to build: exit status 1

It seems that we are building our project in the wrong context. .NET usually has multiple modules and a shared project root. We can tell Build Service to use the src folder as the project root (since it has our .cs_proj file), and also point to a specific .NET project path that we'd like to build:

> kp image save dotnet-eshopweb \
--tag harbor.registry.com/apps/dotnet-eshopweb \
--git https://github.com/dotnet-architecture/eShopOnWeb.git \
--git-revision main \
--sub-path=./src \
--env BP_DOTNET_PROJECT_PATH="./Web"
Patching Image Resource...
Image Resource "dotnet-eshopweb" patched

Let's check out the updated logs:

> kp build logs dotnet-eshopweb
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  + env:
  + - name: BP_DOTNET_PROJECT_PATH
  +   value: ./Web
  resources: {}
  source:
    git:
      revision: 896fae150fbcec5de8a1fb923eab5dfafe09422a
      url: https://github.com/dotnet-architecture/eShopOnWeb.git
  -   subPath: ./src/Web
  +   subPath: ./src
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a"...
Successfully cloned "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a" in path "/workspace"
===> ANALYZE
Previous image with name "harbor.registry.com/apps/dotnet-eshopweb" not found
===> DETECT
7 of 13 buildpacks participating
paketo-buildpacks/ca-certificates    3.0.2
tanzu-buildpacks/dotnet-core-runtime 0.3.1
tanzu-buildpacks/dotnet-core-aspnet  0.3.2
tanzu-buildpacks/dotnet-core-sdk     0.3.1
tanzu-buildpacks/icu                 0.0.13
tanzu-buildpacks/dotnet-publish      0.5.1
tanzu-buildpacks/dotnet-execute      0.4.1
===> RESTORE
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Tanzu .NET Core Runtime Buildpack 0.3.1
  Resolving Dotnet Core Runtime version
    Candidate version sources (in priority order):
      Web.csproj -> "6.0.0"
      <unknown>  -> ""

    Selected dotnet-runtime version (using Web.csproj): 6.0.0

  Executing build process
    Installing Dotnet Core Runtime 6.0.0
      Completed in 4.383s

  Configuring environment for build and launch
    DOTNET_ROOT -> "/workspace/.dotnet_root"

  Configuring environment for build
    RUNTIME_VERSION -> "6.0.0"

Tanzu ASP.NET Core Buildpack 0.3.2
  Resolving Dotnet Core ASPNet version
    Candidate version sources (in priority order):
      RUNTIME_VERSION -> "6.0.0"
      Web.csproj      -> "6.0.0"
      <unknown>       -> ""

    Selected dotnet-aspnetcore version (using RUNTIME_VERSION): 6.0.0

  Executing build process
    Installing Dotnet Core ASPNet 6.0.0
      Completed in 1.253s

  Configuring environment
    DOTNET_ROOT -> "/workspace/.dotnet_root"

Tanzu .NET Core SDK Buildpack 0.3.1
  Resolving .NET Core SDK version
    Candidate version sources (in priority order):
      RUNTIME_VERSION -> "6.0.100"
      <unknown>       -> "*"
      Web.csproj      -> "6.0.*"

    Selected .NET Core SDK version (using RUNTIME_VERSION): 6.0.100

  Executing build process
    Installing .NET Core SDK 6.0.100
      Completed in 16.334s

  Configuring environment
    DOTNET_ROOT -> "/workspace/.dotnet_root"
    PATH        -> "/workspace/.dotnet_root:$PATH"

Tanzu ICU Buildpack 0.0.13
  Executing build process
    Installing ICU
      Completed in 485ms

Tanzu .Net Publish Buildpack 0.5.1
  Executing build process
    Running 'dotnet publish /workspace/Web --configuration Release --runtime ubuntu.18.04-x64 --self-contained false --output /tmp/dotnet-publish-output4061578451'
failed to execute 'dotnet publish': exit status 1
      Failed after 37.399735604s

        Welcome to .NET 6.0!
        ---------------------
        SDK Version: 6.0.100

        Telemetry
        ---------
        The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

        Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

        ----------------
        Installed an ASP.NET Core HTTPS development certificate.
        To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
        Learn about HTTPS: https://aka.ms/dotnet-https
        ----------------
        Write your first app: https://aka.ms/dotnet-hello-world
        Find out what's new: https://aka.ms/dotnet-whats-new
        Explore documentation: https://aka.ms/dotnet-docs
        Report issues and find source on GitHub: https://github.com/dotnet/core
        Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
        --------------------------------------------------------------------------------------
        Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
        Copyright (C) Microsoft Corporation. All rights reserved.

          Determining projects to restore...
          Restored /workspace/BlazorShared/BlazorShared.csproj (in 2.82 sec).
          Restored /workspace/BlazorAdmin/BlazorAdmin.csproj (in 4.12 sec).
          Restored /workspace/ApplicationCore/ApplicationCore.csproj (in 7.8 sec).
          Restored /workspace/Infrastructure/Infrastructure.csproj (in 4.28 sec).
          Restored /workspace/Web/Web.csproj (in 17.21 sec).

          Restore operation started...

          6 libraries restored in 0.96 seconds

          BlazorShared -> /workspace/BlazorShared/bin/Release/net6.0/BlazorShared.dll
          ApplicationCore -> /workspace/ApplicationCore/bin/Release/net6.0/ApplicationCore.dll
          Infrastructure -> /workspace/Infrastructure/bin/Release/net6.0/Infrastructure.dll
          BlazorAdmin -> /workspace/BlazorAdmin/bin/Release/net6.0/BlazorAdmin.dll
          BlazorAdmin (Blazor output) -> /workspace/BlazorAdmin/bin/Release/net6.0/wwwroot

          Bundler: Begin processing bundleconfig.json
          Bundler: Done processing bundleconfig.json
        /workspace/Web/SlugifyParameterTransformer.cs(8,19): warning CS8767: Nullability of reference types in type of parameter 'value' of 'string SlugifyParameterTransformer.TransformOutbound(object value)' doesn't match implicitly implemented member 'string? IOutboundParameterTransformer.TransformOutbound(object? value)' (possibly because of nullability attributes). [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Logout.cshtml.cs(35,64): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(60,42): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(65,69): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/ManageController.cs(31,12): warning CS8618: Non-nullable property 'StatusMessage' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Extensions/UrlHelperExtensions.cs(7,16): warning CS8603: Possible null reference return. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/UserController.cs(27,12): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/HealthChecks/HomePageHealthCheck.cs(22,23): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Manage/ManageNavPages.cs(30,16): warning CS8603: Possible null reference return. [/workspace/Web/Web.csproj]
        /workspace/Web/Services/CatalogItemViewModelService.cs(21,9): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Configuration/RevokeAuthenticationEvents.cs(25,22): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Features/OrderDetails/GetOrderDetailsHandler.cs(29,20): warning CS8603: Possible null reference return. [/workspace/Web/Web.csproj]
        /workspace/Web/Features/MyOrders/GetMyOrdersHandler.cs(31,26): warning CS8601: Possible null reference assignment. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/UserController.cs(31,14): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/ManageController.cs(383,29): warning CS8600: Converting null literal or possible null value to non-nullable type. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/PaginationInfoViewModel.cs(9,19): warning CS8618: Non-nullable property 'Previous' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/PaginationInfoViewModel.cs(10,19): warning CS8618: Non-nullable property 'Next' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/OrderViewModel.cs(15,20): warning CS8618: Non-nullable property 'ShippingAddress' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/OrderItemViewModel.cs(6,19): warning CS8618: Non-nullable property 'ProductName' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/OrderItemViewModel.cs(10,19): warning CS8618: Non-nullable property 'PictureUrl' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ShowRecoveryCodesViewModel.cs(5,21): warning CS8618: Non-nullable property 'RecoveryCodes' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/SetPasswordViewModel.cs(11,19): warning CS8618: Non-nullable property 'NewPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/SetPasswordViewModel.cs(16,19): warning CS8618: Non-nullable property 'ConfirmPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/SetPasswordViewModel.cs(18,19): warning CS8618: Non-nullable property 'StatusMessage' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/RemoveLoginViewModel.cs(5,19): warning CS8618: Non-nullable property 'LoginProvider' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/RemoveLoginViewModel.cs(6,19): warning CS8618: Non-nullable property 'ProviderKey' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/IndexViewModel.cs(7,19): warning CS8618: Non-nullable property 'Username' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/IndexViewModel.cs(13,19): warning CS8618: Non-nullable property 'Email' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/IndexViewModel.cs(17,19): warning CS8618: Non-nullable property 'PhoneNumber' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/IndexViewModel.cs(19,19): warning CS8618: Non-nullable property 'StatusMessage' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/SlugifyParameterTransformer.cs(10,37): warning CS8603: Possible null reference return. [/workspace/Web/Web.csproj]
        /workspace/Web/SlugifyParameterTransformer.cs(13,30): warning CS8604: Possible null reference argument for parameter 'input' in 'string Regex.Replace(string input, string pattern, string replacement)'. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ExternalLoginsViewModel.cs(9,33): warning CS8618: Non-nullable property 'CurrentLogins' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ExternalLoginsViewModel.cs(10,40): warning CS8618: Non-nullable property 'OtherLogins' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ExternalLoginsViewModel.cs(12,19): warning CS8618: Non-nullable property 'StatusMessage' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/EnableAuthenticatorViewModel.cs(13,19): warning CS8618: Non-nullable property 'Code' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/EnableAuthenticatorViewModel.cs(16,19): warning CS8618: Non-nullable property 'SharedKey' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/EnableAuthenticatorViewModel.cs(19,19): warning CS8618: Non-nullable property 'AuthenticatorUri' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ChangePasswordViewModel.cs(10,19): warning CS8618: Non-nullable property 'OldPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ChangePasswordViewModel.cs(16,19): warning CS8618: Non-nullable property 'NewPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ChangePasswordViewModel.cs(21,19): warning CS8618: Non-nullable property 'ConfirmPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Manage/ChangePasswordViewModel.cs(23,19): warning CS8618: Non-nullable property 'StatusMessage' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/File/FileViewModel.cs(5,19): warning CS8618: Non-nullable property 'FileName' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/File/FileViewModel.cs(6,19): warning CS8618: Non-nullable property 'Url' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/File/FileViewModel.cs(7,19): warning CS8618: Non-nullable property 'DataBase64' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/CatalogItemViewModel.cs(6,19): warning CS8618: Non-nullable property 'Name' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/CatalogItemViewModel.cs(7,19): warning CS8618: Non-nullable property 'PictureUri' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/CatalogIndexViewModel.cs(8,39): warning CS8618: Non-nullable property 'CatalogItems' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/CatalogIndexViewModel.cs(9,33): warning CS8618: Non-nullable property 'Brands' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/CatalogIndexViewModel.cs(10,33): warning CS8618: Non-nullable property 'Types' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/CatalogIndexViewModel.cs(13,36): warning CS8618: Non-nullable property 'PaginationInfo' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/ResetPasswordViewModel.cs(9,19): warning CS8618: Non-nullable property 'Email' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/ResetPasswordViewModel.cs(14,19): warning CS8618: Non-nullable property 'Password' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/ResetPasswordViewModel.cs(19,19): warning CS8618: Non-nullable property 'ConfirmPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/ResetPasswordViewModel.cs(21,19): warning CS8618: Non-nullable property 'Code' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/RegisterViewModel.cs(10,19): warning CS8618: Non-nullable property 'Email' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/RegisterViewModel.cs(16,19): warning CS8618: Non-nullable property 'Password' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/RegisterViewModel.cs(21,19): warning CS8618: Non-nullable property 'ConfirmPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/LoginWith2faViewModel.cs(11,19): warning CS8618: Non-nullable property 'TwoFactorCode' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/OrderController.cs(25,62): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/OrderController.cs(25,62): warning CS8604: Possible null reference argument for parameter 'userName' in 'GetMyOrders.GetMyOrders(string userName)'. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/OrderController.cs(33,66): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Controllers/OrderController.cs(33,66): warning CS8604: Possible null reference argument for parameter 'userName' in 'GetOrderDetails.GetOrderDetails(string userName, int orderId)'. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/LoginViewModel.cs(9,19): warning CS8618: Non-nullable property 'Email' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/ViewModels/Account/LoginViewModel.cs(13,19): warning CS8618: Non-nullable property 'Password' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Order/Detail.cshtml(20,68): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Login.cshtml.cs(25,12): warning CS8618: Non-nullable property 'Input' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Login.cshtml.cs(25,12): warning CS8618: Non-nullable property 'ExternalLogins' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Login.cshtml.cs(25,12): warning CS8618: Non-nullable property 'ReturnUrl' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Login.cshtml.cs(25,12): warning CS8618: Non-nullable property 'ErrorMessage' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Manage/TwoFactorAuthentication.cshtml(8,6): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Login.cshtml.cs(46,23): warning CS8618: Non-nullable property 'Email' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Login.cshtml.cs(50,23): warning CS8618: Non-nullable property 'Password' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(24,12): warning CS8618: Non-nullable property 'Input' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(24,12): warning CS8618: Non-nullable property 'ReturnUrl' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/BasketItemViewModel.cs(9,19): warning CS8618: Non-nullable property 'ProductName' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/BasketItemViewModel.cs(16,19): warning CS8618: Non-nullable property 'PictureUrl' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Checkout.cshtml.cs(25,32): warning CS8625: Cannot convert null literal to non-nullable reference type. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(46,23): warning CS8618: Non-nullable property 'Email' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(52,23): warning CS8618: Non-nullable property 'Password' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(57,23): warning CS8618: Non-nullable property 'ConfirmPassword' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Error.cshtml.cs(10,19): warning CS8618: Non-nullable property 'RequestId' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/BasketViewModel.cs(11,19): warning CS8618: Non-nullable property 'BuyerId' must contain a non-null value when exiting constructor. Consider declaring the property as nullable. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Checkout.cshtml.cs(79,82): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Checkout.cshtml.cs(79,82): warning CS8604: Possible null reference argument for parameter 'userName' in 'Task<BasketViewModel> IBasketViewModelService.GetOrCreateBasketForUser(string userName)'. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Shared/Components/Basket/Default.cshtml(12,15): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Areas/Identity/Pages/Account/Register.cshtml.cs(84,91): warning CS8604: Possible null reference argument for parameter 'value' in 'string TextEncoder.Encode(string value)'. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Manage/ShowRecoverCodes.cshtml(19,34): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Checkout.cshtml.cs(92,25): warning CS8601: Possible null reference assignment. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Index.cshtml.cs(64,27): warning CS8600: Converting null literal or possible null value to non-nullable type. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Index.cshtml.cs(66,13): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Index.cshtml.cs(68,20): warning CS8603: Possible null reference return. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Index.cshtml.cs(73,24): warning CS8600: Converting null literal or possible null value to non-nullable type. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Basket/Index.cshtml.cs(79,32): warning CS8600: Converting null literal or possible null value to non-nullable type. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/Components/BasketComponent/Default.cshtml(11,11): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/_editCatalog.cshtml(5,50): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/Components/BasketComponent/Basket.cs(36,63): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/Components/BasketComponent/Basket.cs(36,63): warning CS8604: Possible null reference argument for parameter 'username' in 'Task<int> IBasketViewModelService.CountTotalBasketItems(string username)'. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/Components/BasketComponent/Basket.cs(57,16): warning CS8603: Possible null reference return. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Shared/_LoginPartial.cshtml(1,6): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Manage/EnableAuthenticator.cshtml(23,58): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Manage/ExternalLogins.cshtml(7,6): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/_pagination.cshtml(6,34): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Pages/Shared/_product.cshtml(4,46): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
        /workspace/Web/Views/Manage/MyAccount.cshtml(19,22): warning CS8602: Dereference of a possibly null reference. [/workspace/Web/Web.csproj]
          Web -> /workspace/Web/bin/Release/net6.0/ubuntu.18.04-x64/Web.dll
        /layers/tanzu-buildpacks_dotnet-core-sdk/dotnet-core-sdk/sdk/6.0.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(213,5): error NETSDK1102: Optimizing assemblies for size is not supported for the selected publish configuration. Please ensure that you are publishing a self-contained app. [/workspace/BlazorAdmin/BlazorAdmin.csproj]
ERROR: failed to build: exit status 1

Our build is still failing, but at least the dotnet publish command gives us a very clear hint as to what is going on. In this particular application, we need to set our application to be self-contained. Let's set the .NET build flags to the desired setting. While we're at it, we'll change the verbosity level to normal, so that we'll get more information during the build:

> kp image save dotnet-eshopweb \
--tag harbor.registry.com/apps/dotnet-eshopweb \
--git https://github.com/dotnet-architecture/eShopOnWeb.git \
--git-revision main \
--sub-path=./src \
--env BP_DOTNET_PROJECT_PATH="./Web" \
--env BP_DOTNET_PUBLISH_FLAGS="--verbosity=normal --self-contained=true"

Let's check the updated logs:

> kp build logs dotnet-eshopweb
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  env:
  - name: BP_DOTNET_PROJECT_PATH
    value: ./Web
  + - name: BP_DOTNET_PUBLISH_FLAGS
  +   value: --verbosity=normal --self-contained=true
  resources: {}
  source:
    git:
      revision: 896fae150fbcec5de8a1fb923eab5dfafe09422a
      url: https://github.com/dotnet-architecture/eShopOnWeb.git
    subPath: ./src
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a"...
Successfully cloned "https://github.com/dotnet-architecture/eShopOnWeb.git" @ "896fae150fbcec5de8a1fb923eab5dfafe09422a" in path "/workspace"
===> ANALYZE
Previous image with name "harbor.registry.com/apps/dotnet-eshopweb" not found
===> DETECT
7 of 13 buildpacks participating
paketo-buildpacks/ca-certificates    3.0.2
tanzu-buildpacks/dotnet-core-runtime 0.3.1
tanzu-buildpacks/dotnet-core-aspnet  0.3.2
tanzu-buildpacks/dotnet-core-sdk     0.3.1
tanzu-buildpacks/icu                 0.0.13
tanzu-buildpacks/dotnet-publish      0.5.1
tanzu-buildpacks/dotnet-execute      0.4.1
===> RESTORE
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Tanzu .NET Core Runtime Buildpack 0.3.1
  Resolving Dotnet Core Runtime version
    Candidate version sources (in priority order):
      Web.csproj -> "6.0.0"
      <unknown>  -> ""

    Selected dotnet-runtime version (using Web.csproj): 6.0.0

  Executing build process
    Installing Dotnet Core Runtime 6.0.0
      Completed in 4.814s

  Configuring environment for build and launch
    DOTNET_ROOT -> "/workspace/.dotnet_root"

  Configuring environment for build
    RUNTIME_VERSION -> "6.0.0"

Tanzu ASP.NET Core Buildpack 0.3.2
  Resolving Dotnet Core ASPNet version
    Candidate version sources (in priority order):
      RUNTIME_VERSION -> "6.0.0"
      Web.csproj      -> "6.0.0"
      <unknown>       -> ""

    Selected dotnet-aspnetcore version (using RUNTIME_VERSION): 6.0.0

  Executing build process
    Installing Dotnet Core ASPNet 6.0.0
      Completed in 1.262s

  Configuring environment
    DOTNET_ROOT -> "/workspace/.dotnet_root"

Tanzu .NET Core SDK Buildpack 0.3.1
  Resolving .NET Core SDK version
    Candidate version sources (in priority order):
      RUNTIME_VERSION -> "6.0.100"
      <unknown>       -> "*"
      Web.csproj      -> "6.0.*"

    Selected .NET Core SDK version (using RUNTIME_VERSION): 6.0.100

  Executing build process
    Installing .NET Core SDK 6.0.100
      Completed in 16.227s

  Configuring environment
    DOTNET_ROOT -> "/workspace/.dotnet_root"
    PATH        -> "/workspace/.dotnet_root:$PATH"

Tanzu ICU Buildpack 0.0.13
  Executing build process
    Installing ICU
      Completed in 471ms

Tanzu .Net Publish Buildpack 0.5.1
  Executing build process
    Running 'dotnet publish /workspace/Web --configuration Release --runtime ubuntu.18.04-x64 --output /tmp/dotnet-publish-output1039265788 --verbosity=normal --self-contained=true'
      Completed in 1m3.68120334s

  Removing source code

Tanzu .NET Execute Buildpack 0.4.1
  Assigning launch processes
    web: /workspace/Web --urls http://0.0.0.0:${PORT:-8080}

===> EXPORT
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'tanzu-buildpacks/dotnet-core-runtime:dotnet-core-runtime'
Adding layer 'tanzu-buildpacks/dotnet-core-aspnet:dotnet-core-aspnet'
Adding layer 'tanzu-buildpacks/dotnet-core-sdk:dotnet-env-var'
Adding layer 'tanzu-buildpacks/icu:icu'
Adding layer 'launch.sbom'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving harbor.registry.com/apps/dotnet-eshopweb...
*** Images (sha256:48ac5c34be35b15589edb25a186978b19c70c397c08f618795799b3d65a454a0):
      harbor.registry.com/apps/dotnet-eshopweb
      harbor.registry.com/apps/dotnet-eshopweb:b3.20220116.122508
Adding cache layer 'tanzu-buildpacks/dotnet-core-runtime:dotnet-core-runtime'
Adding cache layer 'tanzu-buildpacks/dotnet-core-aspnet:dotnet-core-aspnet'
Adding cache layer 'tanzu-buildpacks/dotnet-core-sdk:dotnet-core-sdk'
Adding cache layer 'tanzu-buildpacks/icu:icu'
===> COMPLETION
Build successful

This looks A LOT better.

Running the image

Let's see if the image runs as expected:

We still see some errors, but if we'll enable the development mode as hinted on screen, we'll see that the error is as follows:

After some googling, we can see that SQL Server Express is only supported on Windows. This will require us to make some architectural code changes in our app to adapt to a linux world, but that's standard development effort and out of scope for Cloud Native Buildpacks themselves.

Additional tips and tricks

• If your application fails on error NU1605, make sure you update your project file with the following XML entry to force it to use the correct runtime for your desired runtime:

  <PackageReference Include="Microsoft.NETCore.Targets" Version="5.0.0" PrivateAssets="all" />
  

• If you're working at an airgapped environment, you may need to clear the default nuget repositories to force the build to use your private registry only. In your nuget.config:

  <configuration>
  <packageSources>
        <clear /> <!-- ensure only the sources defined below are used -->
        <add key="MyPrivateRepo - ES" value="https://MyPrivateRepo/ES/nuget" />
    </packageSources>
  </configuration>
  

• If you need to set environment variables for the application runtime, add the env varible to the kp image save command, prefixed with BPE_. For example - --env BPE_MY_RUNTIME_VAR=value. Any env variable that starts with BPE will automatically add the environment variables buildpack to the image.

Frontend VueJS Application

We'll begin with a UI application that is using the popular VueJS library. This application presents a frontend for a Todo application. We won't be focusing on the backend so we'll see some "error loading details" in the UI, but the idea is just to see if the image works as expected.

Git repo

The git repo for this application is available at https://github.com/odedia/todo-ui.git

Build command

Let's try to build the code:

> kp image save todo-vuejs \
--tag harbor.registry.com/tap/todo-vuejs \
--git https://github.com/odedia/todo-ui \
--git-revision main
Creating Image Resource...
Image Resource "todo-vuejs" created

Build logs output

Here's the output from the build logs:

> kp build logs todo-vuejs
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  - source: {}
  + source:
  +   git:
  +     revision: 0236707eb0481fce9d3d292f0a1787a7ced5e706
  +     url: https://github.com/odedia/todo-ui
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/odedia/todo-ui" @ "0236707eb0481fce9d3d292f0a1787a7ced5e706"...
Successfully cloned "https://github.com/odedia/todo-ui" @ "0236707eb0481fce9d3d292f0a1787a7ced5e706" in path "/workspace"
===> ANALYZE
Previous image with name "harbor.registry.com/tap/todo-vuejs" not found
===> DETECT
5 of 17 buildpacks participating
paketo-buildpacks/ca-certificates 3.0.2
tanzu-buildpacks/node-engine      0.8.3
tanzu-buildpacks/npm-install      0.5.3
tanzu-buildpacks/node-module-bom  0.1.4
tanzu-buildpacks/npm-start        0.2.3
===> RESTORE
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Tanzu Node Engine Buildpack 0.8.3
  Resolving Node Engine version
    Candidate version sources (in priority order):
      package.json -> ">=10.12.0"
      .nvmrc       -> "11.6.0"
      <unknown>    -> ""

    Selected Node Engine version (using package.json): 16.13.1

  Executing build process
    Installing Node Engine 16.13.1
      Completed in 1.043s

  Configuring build environment
    NODE_ENV     -> "production"
    NODE_HOME    -> "/layers/tanzu-buildpacks_node-engine/node"
    NODE_VERBOSE -> "false"

  Configuring launch environment
    NODE_ENV     -> "production"
    NODE_HOME    -> "/layers/tanzu-buildpacks_node-engine/node"
    NODE_VERBOSE -> "false"

    Writing profile.d/0_memory_available.sh
      Calculates available memory based on container limits at launch time.
      Made available in the MEMORY_AVAILABLE environment variable.

Tanzu NPM Install Buildpack 0.5.3
  Resolving installation process
    Process inputs:
      node_modules      -> "Not found"
      npm-cache         -> "Not found"
      package-lock.json -> "Found"

    Selected NPM build process: 'npm ci'

  Executing build process
    Running 'npm ci --unsafe-perm --cache /layers/tanzu-buildpacks_npm-install/npm-cache'
      Completed in 7.182s

  Configuring launch environment
    NPM_CONFIG_LOGLEVEL -> "error"

  Configuring environment shared by build and launch
    PATH -> "$PATH:/layers/tanzu-buildpacks_npm-install/modules/node_modules/.bin"


Tanzu Node Module Bill of Materials Generator Buildpack 0.1.4
  Resolving CycloneDX Node.js Module version
    Selected CycloneDX Node.js Module version: 3.0.7

  Executing build process
    Installing CycloneDX Node.js Module 3.0.7
      Completed in 42ms

  Configuring environment
    Appending CycloneDX Node.js Module onto PATH

  Running CycloneDX Node.js Module
    Running 'cyclonedx-bom -o bom.json'
      Completed in 501ms

Tanzu NPM Start Buildpack 0.2.3
  Assigning launch processes
    web: node server.js

===> EXPORT
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'tanzu-buildpacks/node-engine:node'
Adding layer 'tanzu-buildpacks/npm-install:modules'
Adding layer 'tanzu-buildpacks/npm-install:npm-cache'
Adding layer 'launch.sbom'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving harbor.registry.com/tap/todo-vuejs...
*** Images (sha256:493cb51b0cd480eb68228da37332a06112141284ee55266faca2047503b99cab):
      harbor.registry.com/tap/todo-vuejs
      harbor.registry.com/tap/todo-vuejs:b1.20220117.104712
Adding cache layer 'tanzu-buildpacks/node-engine:node'
Adding cache layer 'tanzu-buildpacks/npm-install:modules'
Adding cache layer 'tanzu-buildpacks/npm-install:npm-cache'
Adding cache layer 'tanzu-buildpacks/node-module-bom:cyclonedx-node-module'
===> COMPLETION
Build successful

Running the image

Here's the command to run the image:

> docker run -p 8080:8080 harbor.registry.com/tap/todo-vuejs:b1.20220117.104712
Unable to find image 'harbor.registry.com/tap/todo-vuejs:b1.20220117.104712' locally
b1.20220117.104712: Pulling from tap/todo-vuejs
bf99a8b93828: Already exists
1da1131b1360: Already exists
48bde579d8cc: Already exists
1929d0623312: Already exists
1fea93ec33be: Already exists
fed2bcb93d83: Already exists
eb2e57672edf: Already exists
03e598551527: Pull complete
00da406126de: Pull complete
841d6c40c392: Pull complete
b3e12a218902: Pull complete
40d17e1b116e: Pull complete
5baaabcf4fd8: Pull complete
5a44e4f7b58d: Pull complete
Digest: sha256:493cb51b0cd480eb68228da37332a06112141284ee55266faca2047503b99cab
Status: Downloaded newer image for harbor.registry.com/tap/todo-vuejs:b1.20220117.104712
cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory
/layers/tanzu-buildpacks_node-engine/node/profile.d/0_memory_available.sh: line 3: / ( 1024 * 1024 ) : syntax error: operand expected (error token is "/ ( 1024 * 1024 ) ")
server started 5000

Oops - we immediatly see that the container boots on default node port of 5000. This means we can't access the application since nothing maps to 8080. We could just map intenral port 5000 to external port 8080, but conventions are good. We can change the source code, but it's possible we want to keep port 5000 as the default for local development and debugging. The alternative is to set the PORT env varible. The Node buildpack supports embedding launch-time environment varialbes by utilizing the Environment Variables buildpack. Following the documentation, it appears we need to set a value called BPE_PORT:

> kp image save todo-vuejs \
--tag harbor.registry.com/tap/todo-vuejs \
--git https://github.com/odedia/todo-ui \
--git-revision main \
> --env BPE_PORT=8080
Patching Image Resource...
Image Resource "todo-vuejs" patched

We can confirm in the logs that the DETECT phase now contributes the Environment Variables buildpack, which is a good sign:

===> DETECT
6 of 17 buildpacks participating
paketo-buildpacks/ca-certificates       3.0.2
tanzu-buildpacks/node-engine            0.8.3
tanzu-buildpacks/npm-install            0.5.3
tanzu-buildpacks/node-module-bom        0.1.4
tanzu-buildpacks/npm-start              0.2.3
paketo-buildpacks/environment-variables 4.0.1

And indeed, while running the image again, we see the application listening on port 8080:

> docker run -p 8080:8080 harbor.registry.com/tap/todo-vuejs:b2.20220117.105600
Unable to find image 'harbor.registry.com/tap/todo-vuejs:b2.20220117.105600' locally
b2.20220117.105600: Pulling from tap/todo-vuejs
bf99a8b93828: Already exists
1da1131b1360: Already exists
48bde579d8cc: Already exists
1929d0623312: Already exists
1fea93ec33be: Already exists
fed2bcb93d83: Already exists
eb2e57672edf: Already exists
03e598551527: Already exists
00da406126de: Already exists
f9724edbdfed: Pull complete
f4702cd992a7: Pull complete
40d17e1b116e: Pull complete
0dc2cd914cc8: Pull complete
5a44e4f7b58d: Pull complete
Digest: sha256:cdd54580ea9554927a80f7c4ccdbd97046624a6001c137f9c0cecc9487300389
Status: Downloaded newer image for harbor.registry.com/tap/todo-vuejs:b2.20220117.105600
cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory
/layers/tanzu-buildpacks_node-engine/node/profile.d/0_memory_available.sh: line 3: / ( 1024 * 1024 ) : syntax error: operand expected (error token is "/ ( 1024 * 1024 ) ")
server started 8080

Our application is now shown as expected:

Changing the defaults

Does this buildpack really does what we want it to do? Node is a language that can run as a standard backend server, for example to host your APIs, but for frontend applications we really just use it to compile our libraries into standard HTML+Javascript or Typescript. We want Node when doing local development and debugging, but once that part is done - we really just want to take these artifacts and serve them over a web server such as HTTPD or Nginx. Let's see what really happens in our container at the moment:

> docker exec -it b4db4162a405 /bin/bash
cnb@b4db4162a405:/workspace$ ls -l
total 1308
-rwxr-xr-x 1 cnb cnb     521 Jan  1  1980 Dockerfile
-rwxr-xr-x 1 cnb cnb     364 Jan  1  1980 README.md
-rw-r--r-- 1 cnb cnb     716 Jan  1  1980 Tiltfile
-rwxr-xr-x 1 cnb cnb      53 Jan  1  1980 babel.config.js
-rwxr-xr-x 1 cnb cnb      86 Jan  1  1980 build.sh
-rwxr-xr-x 1 cnb cnb      97 Jan  1  1980 buildAndPush.sh
drwxr-sr-x 2 cnb cnb    4096 Jan  1  1980 ci
drwxr-sr-x 2 cnb cnb    4096 Jan  1  1980 config
drwxr-sr-x 4 cnb cnb    4096 Jan  1  1980 dist
-rwxr-xr-x 1 cnb cnb     310 Jan  1  1980 manifest-docker.yml
-rwxr-xr-x 1 cnb cnb     120 Jan  1  1980 manifest-tas4vms.yml
-rwxr-xr-x 1 cnb cnb     120 Jan  1  1980 manifest.yml
lrwxrwxrwx 1 cnb cnb      57 Jan  1  1980 node_modules -> /layers/tanzu-buildpacks_npm-install/modules/node_modules
-rwxr-xr-x 1 cnb cnb 1257671 Jan  1  1980 package-lock.json
-rwxr-xr-x 1 cnb cnb    2508 Jan  1  1980 package.json
drwxr-sr-x 2 cnb cnb    4096 Jan  1  1980 public
-rwxr-xr-x 1 cnb cnb     271 Jan  1  1980 server.js
drwxr-sr-x 3 cnb cnb    4096 Jan  1  1980 src
-rwxr-xr-x 1 cnb cnb      46 Jan  1  1980 vue.config.js

We seem to have a dist folder that contains our desired HTML+Javascript/Typescript, but that's not what the container is currently serving. Checking the package.json file, we can see that the available scripts are:

  "scripts": {
    "serve": "vue-cli-service serve",
    "build": "vue-cli-service build",
    "start": "node server.js",
    "lint": "vue-cli-service lint"
  }

If we'll take a closer look at the build logs from above, the NPM start buildpack contributed the start command as listed in the json:

Tanzu NPM Start Buildpack 0.2.3
  Assigning launch processes
    web: node server.js

So, we're at an odd place - the application is running, but it's probably not optimized as we'd like.

What we really want to do is to take the artifacts in the dist folder and host that as a web server. Sadly, there is no OOTB solution for this yet with existing buildpacks. This is something that would probably change in the near future. In order to solve this issue, we can solve it in two approaches, both are not perfect at the moment:

Option 1: Compile NPM externally

This option would build the source code into the dist folder before building the final image witth TBS. In order to do that, we first need to create two files for NGINX: nginx.conf and mime.types. Both of these files should exist at the root of the repository. mime.types is a standard file that can be replicated wherever needed. A sample file is available here. You may want to tweak nginx.conf per your own use cases, but here's a good starting point:

worker_processes 1;
daemon off;

error_log stderr;
events { worker_connections 1024; }

http {
  charset utf-8;
  log_format cloudfoundry 'NginxLog "$request" $status $body_bytes_sent';
  access_log /dev/stdout cloudfoundry;
  default_type application/octet-stream;
  include mime.types;
  sendfile on;

  tcp_nopush on;
  keepalive_timeout 30;
  port_in_redirect off; # Ensure that redirects don't include the internal container PORT - 8080

  server {
    listen {{ 8080 }};
    root .;
    index index.html;
  }
}

Note that the root folder to serve is the root of the target workspace folder, since we only use the dist folder as the root of our container. Also, We again use port 8080 to host the application.

To get this working, we first need to update the build command in package.json to also copy the two NGINX files to the dist folder:

  "scripts": {
    "serve": "vue-cli-service serve",
    "build": "vue-cli-service build && cp nginx.conf ./dist && cp mime.types ./dist",
    "start": "node server.js",
    "lint": "vue-cli-service lint"
  }

Let's try to run the flow:

npm install && npm run build && kp image save todo-vuejs --tag harbor.registry.com/tap/todo-vuejs --local-path ./dist

The build logs are a lot simpler now - they just server static content behind NGINX:

> kp build logs todo-vuejs
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  - source: {}
  + source:
  +   registry:
  +     image: harbor.registry.com/tap/todo-vuejs-source@sha256:772c0bb4faa607fde8f6139135dd27814cf0a38b4d27e8abdcb68d1f528ab1d5
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Pulling harbor.registry.com/tap/todo-vuejs-source@sha256:772c0bb4faa607fde8f6139135dd27814cf0a38b4d27e8abdcb68d1f528ab1d5...
Successfully pulled harbor.registry.com/tap/todo-vuejs-source@sha256:772c0bb4faa607fde8f6139135dd27814cf0a38b4d27e8abdcb68d1f528ab1d5 in path "/workspace"
===> ANALYZE
===> DETECT
tanzu-buildpacks/nginx 0.4.4
===> RESTORE
Restoring metadata for "tanzu-buildpacks/nginx:nginx" from app image
===> BUILD
Tanzu Nginx Server Buildpack 0.4.4
  Resolving Nginx Server version
    Candidate version sources (in priority order):
      buildpack.toml -> "1.21.*"

    Selected Nginx Server version (using buildpack.toml): 1.21.4

===> EXPORT
Reusing layers from image 'harbor.registry.com/tap/todo-vuejs@sha256:cc57796966f948dbec6531f399cdfbb5a2840730405810efd70eeb205f79a23a'
Reusing layer 'tanzu-buildpacks/nginx:nginx'
Reusing 1/1 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving harbor.registry.com/tap/todo-vuejs...
*** Images (sha256:cc57796966f948dbec6531f399cdfbb5a2840730405810efd70eeb205f79a23a):
      harbor.registry.com/tap/todo-vuejs
      harbor.registry.com/tap/todo-vuejs:b1.20220117.112114
===> COMPLETION
Build successful

Let's see if the application is still running as expected:

> docker run -p 8080:8080 harbor.registry.com/tap/todo-vuejs:b1.20220117.112114
Unable to find image 'harbor.registry.com/tap/todo-vuejs:b1.20220117.112114' locally
b1.20220117.112114: Pulling from tap/todo-vuejs
bf99a8b93828: Already exists
1da1131b1360: Already exists
48bde579d8cc: Already exists
1929d0623312: Already exists
1fea93ec33be: Already exists
5ad05d7a8092: Pull complete
3499fa7eadde: Pull complete
40d17e1b116e: Pull complete
274f229843b4: Pull complete
5a44e4f7b58d: Pull complete
Digest: sha256:cc57796966f948dbec6531f399cdfbb5a2840730405810efd70eeb205f79a23a
Status: Downloaded newer image for harbor.registry.com/tap/todo-vuejs:b1.20220117.112114

Our application is exposed as expected:

So things work, great! But, is this a good practice? probably not.

• We need to compile our code outside the container.
• The NPM version that runs on the target system can impact our build. Changes to the hosted environment can produce different results.
• We need to send source code from a local path to Build Service. This means that we are basically decoupled from our git repository. Changes in git will not be reflected in the build pipeline.
• We create dependencies between Node buildpack and the NGINX buildpack - our build command needs to copy files to the dist folder.
• In general, this feels very "scripty" and you might as well just revert to a Dockerfile at this point (although you always gain the benefits of proactive CVE patching for your base OS stack and middleware, even with this approach).

Option 2: Create a custom buildpack

We can take a different approach and just tell Build Service to run the Node Buildpack followed by the NGINX buildpack. This way we'll get better automation and can still target a git repository and not a local folder.

The first thing we need is to know how our buildpacks are named. We can use the following command to get information about our default clusterbuilder, which is the one we used up to this point:

> kp clusterbuilder status default
Status:       Ready
Image:        harbor.registry.com/build-service/tbs:clusterbuilder-default@sha256:e2f4db390a617d11a475a5ebbed4bd17ddd9375a5e1b9f489621480ae58dcc43
Stack ID:     io.buildpacks.stacks.bionic
Run Image:    harbor.registry.com/build-service/tbs@sha256:b69b4f654e6a2d2d69d1cca1a51638401127b6fb3556faa6304cab1575ef7c16

Stack Ref:
  Name:       base
  Kind:       ClusterStack
Store Ref:
  Name:       default
  Kind:       ClusterStore

BUILDPACK ID                                    VERSION    HOMEPAGE
paketo-buildpacks/bellsoft-liberica             9.0.2      https://github.com/paketo-buildpacks/bellsoft-liberica
tanzu-buildpacks/go-dist                        0.6.5
tanzu-buildpacks/cpython                        0.2.0      https://github.com/pivotal-cf/tanzu-cpython
tanzu-buildpacks/dotnet-core-sdk                0.3.1
tanzu-buildpacks/node-engine                    0.8.3
tanzu-buildpacks/apache-skywalking              4.1.2      https://github.com/pivotal-cf/tanzu-apache-skywalking
tanzu-buildpacks/dotnet-core-runtime            0.3.1
tanzu-buildpacks/appdynamics                    4.1.4      https://github.com/pivotal-cf/tanzu-appdynamics
paketo-buildpacks/gradle                        6.0.2      https://github.com/paketo-buildpacks/gradle
tanzu-buildpacks/jprofiler                      4.0.2      https://github.com/pivotal-cf/tanzu-jprofiler
tanzu-buildpacks/miniconda                      0.1.4      https://github.com/pivotal-cf/tanzu-miniconda
tanzu-buildpacks/new-relic                      4.0.4      https://github.com/pivotal-cf/tanzu-new-relic
tanzu-buildpacks/overops                        4.0.3      https://github.com/pivotal-cf/tanzu-overops
tanzu-buildpacks/pipenv                         0.2.4      https://github.com/pivotal-cf/tanzu-pipenv
tanzu-buildpacks/dotnet-core-aspnet             0.3.2
paketo-buildpacks/maven                         6.1.0      https://github.com/paketo-buildpacks/maven
paketo-buildpacks/apache-tomcat                 7.0.3      https://github.com/paketo-buildpacks/apache-tomcat
tanzu-buildpacks/icu                            0.0.13
paketo-buildpacks/azure-application-insights    5.1.2      https://github.com/paketo-buildpacks/azure-application-insights
tanzu-buildpacks/jrebel                         4.0.3      https://github.com/pivotal-cf/tanzu-jrebel
paketo-buildpacks/sbt                           6.1.0      https://github.com/paketo-buildpacks/sbt
paketo-buildpacks/sbt                           6.0.4      https://github.com/paketo-buildpacks/sbt
tanzu-buildpacks/riverbed                       4.0.2      https://github.com/pivotal-cf/tanzu-riverbed
tanzu-buildpacks/yourkit                        4.0.3      https://github.com/pivotal-cf/tanzu-yourkit
tanzu-buildpacks/nginx                          0.4.4      https://docs.pivotal.io/tanzu-buildpacks/nginx/nginx-buildpack.html
tanzu-buildpacks/pip                            0.6.3      https://github.com/pivotal-cf/tanzu-pip
tanzu-buildpacks/contrast-security              4.1.1      https://github.com/pivotal-cf/tanzu-contrast-security
tanzu-buildpacks/elastic-apm                    4.1.2      https://github.com/pivotal-cf/tanzu-elastic-apm
paketo-buildpacks/maven                         6.0.2      https://github.com/paketo-buildpacks/maven
paketo-buildpacks/google-stackdriver            5.1.2      https://github.com/paketo-buildpacks/google-stackdriver
tanzu-buildpacks/elastic-apm                    4.1.3      https://github.com/pivotal-cf/tanzu-elastic-apm
tanzu-buildpacks/dep                            0.1.4
paketo-buildpacks/syft                          1.3.1      https://github.com/paketo-buildpacks/syft
tanzu-buildpacks/yarn                           0.3.3
tanzu-buildpacks/dynatrace                      4.0.2      https://github.com/pivotal-cf/tanzu-dynatrace
paketo-buildpacks/spring-boot                   5.3.0      https://github.com/paketo-buildpacks/spring-boot
tanzu-buildpacks/jacoco                         4.0.2      https://github.com/pivotal-cf/tanzu-jacoco
tanzu-buildpacks/node-module-bom                0.1.4
tanzu-buildpacks/checkmarx                      4.0.2      https://github.com/pivotal-cf/tanzu-checkmarx
paketo-buildpacks/watchexec                     2.0.2      https://github.com/paketo-buildpacks/watchexec
paketo-buildpacks/upx                           3.0.1      https://github.com/paketo-buildpacks/upx
tanzu-buildpacks/synopsys                       4.0.2      https://github.com/pivotal-cf/tanzu-synopsys
paketo-buildpacks/ca-certificates               3.0.2      https://github.com/paketo-buildpacks/ca-certificates
paketo-buildpacks/procfile                      5.0.2      https://github.com/paketo-buildpacks/procfile
paketo-buildpacks/encrypt-at-rest               4.0.2      https://github.com/paketo-buildpacks/encrypt-at-rest
paketo-buildpacks/leiningen                     4.0.2      https://github.com/paketo-buildpacks/leiningen
paketo-buildpacks/clojure-tools                 2.0.2      https://github.com/paketo-buildpacks/clojure-tools
paketo-buildpacks/syft                          1.4.0      https://github.com/paketo-buildpacks/syft
tanzu-buildpacks/snyk                           4.0.2      https://github.com/pivotal-cf/tanzu-snyk
tanzu-buildpacks/go-build                       0.3.2
tanzu-buildpacks/dotnet-publish                 0.5.1
tanzu-buildpacks/dotnet-execute                 0.4.1
tanzu-buildpacks/pipenv-install                 0.1.6      https://github.com/pivotal-cf/tanzu-pipenv-install
tanzu-buildpacks/npm-install                    0.5.3
tanzu-buildpacks/yarn-install                   0.3.2
tanzu-buildpacks/dep-ensure                     0.1.3
tanzu-buildpacks/go-mod-vendor                  0.2.3
tanzu-buildpacks/pip-install                    0.2.2      https://github.com/pivotal-cf/tanzu-pip-install
tanzu-buildpacks/conda-env-update               0.1.4      https://github.com/pivotal-cf/tanzu-conda-env-update
tanzu-buildpacks/node-run-script                0.2.2
tanzu-buildpacks/npm-start                      0.2.3
tanzu-buildpacks/yarn-start                     0.2.3
tanzu-buildpacks/python-start                   0.4.2      https://github.com/pivotal-cf/tanzu-python-start
tanzu-buildpacks/node-start                     0.3.3
paketo-buildpacks/native-image                  5.0.2      https://github.com/paketo-buildpacks/native-image
paketo-buildpacks/executable-jar                6.0.2      https://github.com/paketo-buildpacks/executable-jar
tanzu-buildpacks/aspectj                        4.0.2      https://github.com/pivotal-cf/tanzu-aspectj
paketo-buildpacks/dist-zip                      5.0.2      https://github.com/paketo-buildpacks/dist-zip
paketo-buildpacks/environment-variables         4.0.1      https://github.com/paketo-buildpacks/environment-variables
paketo-buildpacks/environment-variables         4.0.2      https://github.com/paketo-buildpacks/environment-variables
paketo-buildpacks/image-labels                  4.0.2      https://github.com/paketo-buildpacks/image-labels
tanzu-buildpacks/java                           6.4.0      https://github.com/pivotal-cf/tanzu-java
tanzu-buildpacks/java-native-image              6.5.0      https://github.com/pivotal-cf/tanzu-java-native-image
tanzu-buildpacks/nodejs                         1.13.5     https://docs.pivotal.io/tanzu-buildpacks/nodejs/nodejs-buildpack.html
tanzu-buildpacks/dotnet-core                    1.11.1     https://docs.pivotal.io/tanzu-buildpacks/dotnet-core/dotnet-core-buildpack.html
tanzu-buildpacks/go                             1.10.4     https://docs.pivotal.io/tanzu-buildpacks/go/go-buildpack.html
tanzu-buildpacks/python                         1.1.4


DETECTION ORDER
Group #1
  tanzu-buildpacks/dotnet-core@1.11.1
Group #2
  tanzu-buildpacks/nodejs@1.13.5
Group #3
  tanzu-buildpacks/go@1.10.4
Group #4
  tanzu-buildpacks/python@1.1.4
Group #5
  tanzu-buildpacks/nginx@0.4.4
Group #6
  tanzu-buildpacks/java-native-image@6.5.0
Group #7
  tanzu-buildpacks/java@6.4.0
Group #8
  paketo-buildpacks/procfile@5.0.2

We see that the buildpacks we need are tanzu-buildpacks/nodejs and tanzu-buildpacks/nginx

Now, we need to define a new builder. Let's call it webserver:

> kp builder create webserver --tag harbor.registry.com/apps/webserver --buildpack tanzu-buildpacks/nodejs --buildpack tanzu-buildpacks/nginx
Builder "webserver" created

Before we can update our image, we need to setup the source code to be more aligned with our new folder structure. Our resulting image will have all the NodeJS artifacts and also the target dist folder. Because of that, we need to do the following:

• The npm build command doesn't need to copy the two Nginx artifacts to the dist folder anymore:

  "scripts": {
    "serve": "vue-cli-service serve",
    "build": "vue-cli-service build",
    "start": "node server.js",
    "lint": "vue-cli-service lint"
  }
  

• The root of nginx.conf should now target the dist folder:

worker_processes 1;
daemon off;

error_log stderr;
events { worker_connections 1024; }

http {
  charset utf-8;
  log_format cloudfoundry 'NginxLog "$request" $status $body_bytes_sent';
  access_log /dev/stdout cloudfoundry;
  default_type application/octet-stream;
  include mime.types;
  sendfile on;

  tcp_nopush on;
  keepalive_timeout 30;
  port_in_redirect off; # Ensure that redirects don't include the internal container PORT - 8080

  server {
    listen {{ 8080 }};
    root dist;
    index index.html;
  }
}

With our source code setup as we want, we can initiate the build command:

> kp image save todo-vuejs \
--tag harbor.registry.com/tap/todo-vuejs \
--git https://github.com/odedia/todo-ui \
--git-revision main \
--builder webserver
Patching Image Resource...
Image Resource "todo-vuejs" patched

Let's check the updated logs. We now see that the code is first being built with NPM, and then served via NGINX, which is exactly what we want:

> kp build logs todo-vuejs
===> PREPARE
Build reason(s): CONFIG
CONFIG:
  resources: {}
  source:
  -   registry:
  -     image: harbor.registry.com/tap/todo-vuejs-source@sha256:8783abc3754540ca6c5cb0cca9b92ab5a1bca98a539d7a7473abf3bd53e9a328
  +   git:
  +     revision: 988e7e70f27a2fcb9f9edf2025f1e517cfdef86d
  +     url: https://github.com/odedia/todo-ui
Loading secret for "harbor.registry.com" from secret "registry-credentials" at location "/var/build-secrets/registry-credentials"
Cloning "https://github.com/odedia/todo-ui" @ "988e7e70f27a2fcb9f9edf2025f1e517cfdef86d"...
Successfully cloned "https://github.com/odedia/todo-ui" @ "988e7e70f27a2fcb9f9edf2025f1e517cfdef86d" in path "/workspace"
===> ANALYZE
===> DETECT
6 of 18 buildpacks participating
paketo-buildpacks/ca-certificates 3.0.2
tanzu-buildpacks/node-engine      0.8.3
tanzu-buildpacks/npm-install      0.5.3
tanzu-buildpacks/node-module-bom  0.1.4
tanzu-buildpacks/npm-start        0.2.3
tanzu-buildpacks/nginx            0.4.4
===> RESTORE
Restoring metadata for "tanzu-buildpacks/nginx:nginx" from app image
===> BUILD

Paketo CA Certificates Buildpack 3.0.2
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Tanzu Node Engine Buildpack 0.8.3
  Resolving Node Engine version
    Candidate version sources (in priority order):
      package.json -> ">=10.12.0"
      .nvmrc       -> "11.6.0"
      <unknown>    -> ""

    Selected Node Engine version (using package.json): 16.13.1

  Executing build process
    Installing Node Engine 16.13.1
      Completed in 1.076s

  Configuring build environment
    NODE_ENV     -> "production"
    NODE_HOME    -> "/layers/tanzu-buildpacks_node-engine/node"
    NODE_VERBOSE -> "false"

  Configuring launch environment
    NODE_ENV     -> "production"
    NODE_HOME    -> "/layers/tanzu-buildpacks_node-engine/node"
    NODE_VERBOSE -> "false"

    Writing profile.d/0_memory_available.sh
      Calculates available memory based on container limits at launch time.
      Made available in the MEMORY_AVAILABLE environment variable.

Tanzu NPM Install Buildpack 0.5.3
  Resolving installation process
    Process inputs:
      node_modules      -> "Not found"
      npm-cache         -> "Not found"
      package-lock.json -> "Found"

    Selected NPM build process: 'npm ci'

  Executing build process
    Running 'npm ci --unsafe-perm --cache /layers/tanzu-buildpacks_npm-install/npm-cache'
      Completed in 6.354s

  Configuring launch environment
    NPM_CONFIG_LOGLEVEL -> "error"

  Configuring environment shared by build and launch
    PATH -> "$PATH:/layers/tanzu-buildpacks_npm-install/modules/node_modules/.bin"


Tanzu Node Module Bill of Materials Generator Buildpack 0.1.4
  Resolving CycloneDX Node.js Module version
    Selected CycloneDX Node.js Module version: 3.0.7

  Executing build process
    Installing CycloneDX Node.js Module 3.0.7
      Completed in 46ms

  Configuring environment
    Appending CycloneDX Node.js Module onto PATH

  Running CycloneDX Node.js Module
    Running 'cyclonedx-bom -o bom.json'
      Completed in 496ms

Tanzu NPM Start Buildpack 0.2.3
  Assigning launch processes
    web: node server.js

Tanzu Nginx Server Buildpack 0.4.4
  Resolving Nginx Server version
    Candidate version sources (in priority order):
      buildpack.toml -> "1.21.*"

    Selected Nginx Server version (using buildpack.toml): 1.21.4

===> EXPORT
Reusing layers from image 'harbor.registry.com/tap/todo-vuejs@sha256:6f435c808b0b7b413b7314aeb24bb01f23c859aec27d482c8574bce2435115a3'
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'tanzu-buildpacks/node-engine:node'
Adding layer 'tanzu-buildpacks/npm-install:modules'
Adding layer 'tanzu-buildpacks/npm-install:npm-cache'
Reusing layer 'tanzu-buildpacks/nginx:nginx'
Adding layer 'launch.sbom'
Adding 1/1 app layer(s)
Reusing layer 'launcher'
Adding layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving harbor.registry.com/tap/todo-vuejs...
*** Images (sha256:c8201b2c55727cd0a6ae238a9bb3f889b5cf1247b75b7b1e029c5547a8237db9):
      harbor.registry.com/tap/todo-vuejs
      harbor.registry.com/tap/todo-vuejs:b3.20220117.113942
Adding cache layer 'tanzu-buildpacks/node-engine:node'
Adding cache layer 'tanzu-buildpacks/npm-install:modules'
Adding cache layer 'tanzu-buildpacks/npm-install:npm-cache'
Adding cache layer 'tanzu-buildpacks/node-module-bom:cyclonedx-node-module'
===> COMPLETION
Build successful

Let's verify everything works as expected:

> docker run harbor.registry.com/tap/todo-vuejs:b3.20220117.113942
Unable to find image 'harbor.registry.com/tap/todo-vuejs:b3.20220117.113942' locally
b3.20220117.113942: Pulling from tap/todo-vuejs
bf99a8b93828: Already exists
1da1131b1360: Already exists
48bde579d8cc: Already exists
1929d0623312: Already exists
1fea93ec33be: Already exists
fed2bcb93d83: Already exists
eb2e57672edf: Already exists
03e598551527: Already exists
f76b3557931f: Pull complete
5ad05d7a8092: Pull complete
841d6c40c392: Pull complete
697988c1248a: Pull complete
40d17e1b116e: Pull complete
239731ce13f5: Pull complete
5a44e4f7b58d: Pull complete
Digest: sha256:c8201b2c55727cd0a6ae238a9bb3f889b5cf1247b75b7b1e029c5547a8237db9
Status: Downloaded newer image for harbor.registry.com/tap/todo-vuejs:b3.20220117.113942
cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory
/layers/tanzu-buildpacks_node-engine/node/profile.d/0_memory_available.sh: line 3: / ( 1024 * 1024 ) : syntax error: operand expected (error token is "/ ( 1024 * 1024 ) ")

The application still works as expected:

In the docker logs, we can confirm that it's NGINX hosting the content:

NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET /js/app.8278f7ef.js HTTP/1.1" 200 10484
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771
NginxLog "GET / HTTP/1.1" 200 771

Great! Everything works. However, there is still a small thing that bugs me: Do I really need the entire NPM stack in the target image? This includes the bloated node_modules folder and other artifacts. The size of this new image is around 210mb, while the size of the NGINX-only image is 102mb.

Still, this is probably the recommended way to build frontend applications with Build Service, and I believe this area will improve significantly soon.

This concludes part 1 of this tutorial. In the next one, we'll dive deep into other buildpacks such as Spring Native (Powered by GraalVM), Python applications and some other extreme use cases. Stay tuned!

Kubernetes is the new IaaS.

Whether you’re developing custom applications or purchasing off-the-shelf software from a vendor, the new, modern target runtime are containers, and the container runtime of choice is Kubernetes.

In a rare alignment of the stars for the tech industry, virtually all vendors, private cloud and public cloud providers, are in agreement on a spec and on an API. That's really something. It doesn't have to be the best API in world, or even the most readable, and it doesn’t matter that interaction is mostly done through a wall of YAML. The important takeaway is that it's a language and syntax that almost everyone agreed upon.

But just because the industry agrees on a fundamental, low-level API, doesn't mean that developers should use it. In fact, it probably means the opposite.

The abstraction gap

How often do you develop cloud-based, server side software in C these days? When was the last time you've written something in assembly, if ever? Do you use low-level networking APIs or do you opt for HTTP or gRPC?

We write at higher-level abstractions because they provide us with meaningful benefits:

• Our code is more readable
• We stand on the shoulders of giants who simplified and encapsulated very complicated systems for us
• An abstraction usually provides guardrails. How many times did the type system of Java, or its automated garbage collection, saved you from disaster?
• Underlying low-level APIs may change for better optimization, while our code remains the same

The same can and should be said for Kubernetes. If we truly treat Kubernetes as commodity infrastructure, why do we use it directly? How many developers do you know that have ever seen a vCenter console, or got full admin rights to an AWS console? Hardly any, and for a very good reason.

Developers want to write their business logic and have it run on a Kubernetes cluster.

How? Good question.

Sure, developers need to understand how things work in Kubernetes, and may need to dig deeper into an environment when issues arise, but in their day to day they have so many other things to worry about - business logic, code path optimizations, writing unit tests, merge conflicts, evaluating sources and targets of data - that having to worry about infrastructure is just too much of an overhead. Non-functional requirements can take up a considerable part of a software engineer's time. That is time that could have been spent providing value to the organization with new features, that ultimately provide a better user experience, and eventually more revenue.

A platform that is focused on developer productivity, if done correctly, can remove much of that toil from the day-to-day of a software engineer, otherwise they may spend a large portion of their time building an internal DIY platform.

Let's build an app

In order to fully understand a problem domain, I always like to use a real example. Let's test the hypothesis with a very simple hello world application, written in Java using Spring Boot:

@RestController
class HelloController {
    @Value("${app.message:Hello world!}")
    private String message;

    @GetMapping(value = "/", produces = MediaType.TEXT_PLAIN_VALUE)
    String greeting() {
        // Just return a simple String.
        return message;
    }
}

Simple enough.

Let's add a unit test to make sure our application behaves as expected:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class ApplicationTests {
    @Autowired
    private TestRestTemplate webClient;

    @Test
    public void testGreeting() {
        assertThat(webClient.getForObject("/", String.class)).isEqualTo("Hello world!");
    }
}

Great! Our application is now production ready. 😉

Well, not really. We want to run this application in a production-ready environment, preferably as a container so we can gain better isolation than just a JVM runtime, and also to easily scale out horizontally when needed.

Building the OCI image

First of all, we need to containerize our application. But how? The most obvious way to achieve this is to write a Dockerfile. After some googling, we may end up with this example from the Spring guides website:

FROM openjdk:8-jdk-alpine
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

Now, this will work, but only if we compile and package our code in advance. Let's deconstruct this Dockerfile line by line:

• FROM openjdk:8-jdk-alpine uses a base image from Dockerhub that appears to be based on Alpine Linux and OpenJDK 8. Being bound to JDK 8 is pretty old here at the dawn of 2022, so we may need to google for a better baseline that supports Java 11 or 17.
• ARG JAR_FILE=target/*.jar declares an env variable that contains all the jar files in the target folder. We assume there's only one, or else the next command will not work properly.
• COPY ${JAR_FILE} app.jar copies the jar file to the root of the container folder structure and renames it to app.jar.
• ENTRYPOINT ["java","-jar","/app.jar"] sets the start command for this container - basically run the uber jar using Java. Note the utter lack of any JAVA_OPTS, tweaking for production or changes to the garbage collection settings. We’re using whatever the defaults are, which are probably not optimized for container runtimes.

Let's build the image:

> docker build -t hello-world:v1 .
...
...
 => [stage-1 1/5] FROM docker.io/library/adoptopenjdk:11-jre-hotspot@sha256:ad6431b2e2a6f8016aa6d79c3f588783af9fdc06cafe131fd0d3faf560914b13         9.8s
 => [internal] load build context                                                                                                                    0.0s
 => => transferring context: 232.93kB                                                                                                                0.0s
 => [stage-1 2/5] RUN useradd -m -s /bin/bash app                                                                                                    0.5s
 => [builder 2/6] RUN mkdir /build                                                                                                                   0.4s
 => [builder 3/6] ADD . /build                                                                                                                       0.1s
 => [builder 4/6] WORKDIR /build                                                                                                                     0.0s
 => [builder 5/6] RUN ./mvnw -B clean package                                                                                                      101.9s
 => [builder 6/6] RUN mkdir -p target/dependency && (cd target/dependency; jar -xf ../*.jar)                                                         0.6s
 => [stage-1 3/5] COPY --from=BUILDER /build/target/dependency/BOOT-INF/lib /app/lib                                                                 0.1s
 => [stage-1 4/5] COPY --from=BUILDER /build/target/dependency/META-INF /app/META-INF                                                                0.0s
 => [stage-1 5/5] COPY --from=BUILDER /build/target/dependency/BOOT-INF/classes /app                                                                 0.0s
 => exporting to image                                                                                                                               0.1s
 => => exporting layers                                                                                                                              0.1s
 => => writing image sha256:1433c6a737757449f2697cee31806f69cb0ffac1ca9ea53acae7153e1868d3f2                                                         0.0s
 => => naming to docker.io/library/hello-world:v1                                                                                                    0.0s

So, this was fairly simple, but also quite script-driven. We basically described a step by step of the how, not the what.

As developers, we are now also responsible for 100% of that how. We need to start thinking about security implications. For example, we probably only want to run our container with user privileges. The same Spring blog post recommends to add a custom user and group and run the container with that user:

FROM openjdk:8-jdk-alpine
RUN addgroup -S spring && adduser -S spring -G spring
USER spring:spring
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

Once again, since we as developers are now responsible for 100% of the how, we need to start thinking about optimizations. The sample above takes our large uber jar that can easily reach 150mb or more, and put it in a single layer in the OCI image. What does this mean in practice? It means that for every new build, even for the simplest code change in our custom code, a new 150mb layer (read: tarball) will be created and uploaded to our OCI registry. Later, when pulling the image to our local machines or to a Kubernetes cluster, a 150mb layer will need to be pulled as well. This will happen on every single code change and rebuild of the image. In reality, only a very small sliver of compiled code has changed. Dockerfile treats every new line as its own layer, so it would make a lot more sense to put the third party dependencies in their own layer and have our custom code in its own layer. That way, only a very small layer will need to be uploaded and later pulled. The Spring blog presents the following approach to basically "unzip" the jar file:

FROM openjdk:8-jdk-alpine
RUN addgroup -S spring && adduser -S spring -G spring
USER spring:spring
ARG DEPENDENCY=target/dependency
COPY ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY ${DEPENDENCY}/META-INF /app/META-INF
COPY ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","hello.HelloController"]

Of course, this sort of Dockerfile is only good if you have a previous step that already compiled and packaged the jar for you, and that's usually not what you want. It would be ideal that the compilation of the source code would be done in a containerized environment as well, to remove any risk of an unknown (and potentially insecure) build environment.

We also took some time to build our image with what we consider good security best practices today, but new CVEs are discovered every day, and patches are released to various Linux distributions, to the JDK and to many frameworks you may be using in your code (hi there, log4j 👋). Since we as developers now need to care about the how, it is our responsibility (or a specific person in the organization) to keep things patched continuously.

Cloud Native Buildpacks to the rescue

Luckily, at least in the world of Spring Boot, we have you covered.

Thanks to Cloud-Native Buildpacks CNCF project, we can create images from source code just by saying what we want, without having to explain the how. Cloud Native Buildpacks are available for various modern languages, and of course for Spring Boot. In fact, the Spring Boot maven plugin provides a command just for that:

mvn spring-boot:build-image

This simple command will build an optimized, production-ready, secure image. I may not care how, although I could dive into the spec if I want to understand how it performs the build lifecycle. I can also override the recommended defaults if I wanted to (for example to change the JDK version from the current default of JDK 11 to an older JDK 8 or a newer JDK 17). Lastly - the generated image is created with sensible defaults such as proper segregation of layers in the OCI image so it would be optimized for storage and pulls later on, and an optimized java start command that is recommend for running in a containerized environment. The important piece here is that I delegated some of the responsibility of my non-functional requirements to a platform. In the case of a simple mvn spring-boot:build-image command, it will go out and build the image using the latest patched layers that are available for the OS and OpenJDK. In case of other platforms that support Cloud Native Buildpacks, such as kpack or its commercial flavor Tanzu Build Service, the builds would be triggered automatically on git commit or when an operator uploads updated buildpacks or OS updates (called: stacks ) to the environment.

So, we have a production-ready image. Great! We can now just deploy to Kubernetes and forget about it.

Well... I guess we still have some things we need to do.

Let's try to get as far as we can with just barebones Kubernetes.

Creating a deployment

We'll begin by tagging our built image and pushing it to a docker registry.

docker tag hello-world my-registry.com/apps/hello-world:v1
docker push my-registry.com/apps/hello-world:v1

Now we can go ahead and create a deployment in our Kubernetes cluster. It is considered a best practice to make changes in Kubernetes in a declarative way, so we can keep our configuration in a versioned repository, thus promoting Infrastructure as Code. Let's use imperative commands to generate the initial declarative yaml skeleton we will need:

kubectl create deployment hello-world --image=my-registry.com/apps/hello-world:v1 --dry-run=client -o yaml > deployment.yaml

The result is a good starting point to describe our deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: hello-world
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: hello-world
    spec:
      containers:
      - image: my-registry.com/apps/hello-world:v1
        name: hello-world
        resources: {}
status: {}

We would probably want to change some settings here though. For example, set replica count to 3. We would also need to understand the CPU and memory requirements of our application and set proper Kubernetes resource requests and limits (however, it's best to understand what these settings actually mean). Let's start with an initial settings of 1 core and 384mb of memory, and a max of 1.5 cores and 512mb of memory:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: hello-world
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: hello-world
    spec:
      containers:
      - image: my-registry.com/apps/hello-world:v1
        name: hello-world
        resources:
          requests:
            memory: "384Mi"
            cpu: "1000m"
          limits:
            memory: "512Mi"
            cpu: "1500m"

You would probably also want to define the deployment strategy, as this could have implications on the availability of your app. How many replicas can be unavailable at any given time? How many replicas can be updated simultaneously?

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-world
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  strategy:
    type: RollingUpdate
    rollingUpdate:
       maxSurge: 25%
       maxUnavailable: 25%  
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - image: my-registry.com/apps/hello-world:v1
        name: hello-world
        resources:
          requests:
            memory: "384Mi"
            cpu: "1000m"
          limits:
            memory: "512Mi"
            cpu: "1500m"

We're almost there!

Oh wait. How do I tell Kubernetes where to place each replica? I would like to have a nice spread between available worker nodes for better availability. The below example adds an anti-affinity rule to our deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-world
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  strategy:
    type: RollingUpdate
    rollingUpdate:
       maxSurge: 25%
       maxUnavailable: 25%  
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - hello-world
            topologyKey: "kubernetes.io/hostname"
      containers:
      - image: my-registry.com/apps/hello-world:v1
        name: hello-world
        resources:
          requests:
            memory: "384Mi"
            cpu: "1000m"
          limits:
            memory: "512Mi"
            cpu: "1500m"

This is getting quite long, isn't it? Oh well, whatever it takes to run our simple hello world app on Kubernetes 🤦🏻‍♂️

We can finally deploy our application:

> kubectl apply -f deployment.yaml

Let's make sure our app is working as expected:

> kubectl get pods
NAME                           READY   STATUS             RESTARTS   AGE
hello-world-64bccd8549-fqmxv   0/1     CrashLoopBackOff   3          79s
hello-world-64bccd8549-fxgrr   0/1     CrashLoopBackOff   3          79s
hello-world-64bccd8549-rbw2b   0/1     CrashLoopBackOff   3          79s

Oops. What happened? Let's check the logs:

> kubectl logs --selector app=hello-world
Setting Active Processor Count to 4
unable to calculate memory configuration
fixed memory regions require 596880K which is greater than 512M available for allocation: -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=84880K, -XX:ReservedCodeCacheSize=240M, -Xss1M * 250 threads
ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator': exit status 1

Seems like our resource requests and limits were too low. Until Spring Native is officially GAed with much lower memory footprint requirements and faster startup times, we're still bound to our memory-hungry JVM. Still, this is a reminder that it is our responsibility to tweak the deployment configuration to a production-ready state. After changing the deployment yaml to a limit of 768Mi, we can try again:

> kubectl apply -f deployment.yaml
deployment.apps/hello-world configured


> kubectl get pods
NAME                           READY   STATUS             RESTARTS   AGE
hello-world-5ff7bfb9ff-wgnj6   0/1     Pending            0          38s
hello-world-64bccd8549-fqmxv   0/1     CrashLoopBackOff   5          5m20s
hello-world-64bccd8549-fxgrr   0/1     CrashLoopBackOff   5          5m20s
hello-world-64bccd8549-rbw2b   0/1     CrashLoopBackOff   5          5m20s

Hmm... seems like our rollout is stuck in Pending state. We probably didn't take into consideration the correct relationship between our RollingUpdate strategy, the Node Anti-Affinity rules and the number of available worker nodes, so we are stuck in a form of deadlock. Again, it is your responsibility to take these matters into consideration.

Let's change our deployment.yaml to allow for up to 50% of the replicas to be unavailable. This may or may not be an acceptable solution in your production environment. The final deployment.yaml now looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-world
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  strategy:
    type: RollingUpdate
    rollingUpdate:
       maxSurge: 25%
       maxUnavailable: 50%  
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - hello-world
            topologyKey: "kubernetes.io/hostname"
      containers:
      - image: my-registry.com/apps/hello-world:v1
        name: hello-world
        resources:
          requests:
            memory: "384Mi"
            cpu: "1000m"
          limits:
            memory: "768Mi"
            cpu: "1500m"

Let's check the results:

> kubectl apply -f deployment.yaml 
deployment.apps/hello-world configured

> kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
hello-world-5ff7bfb9ff-mph2v   1/1     Running   0          99s
hello-world-5ff7bfb9ff-pmkxx   1/1     Running   0          95s
hello-world-5ff7bfb9ff-xrtdx   1/1     Running   0          99s

Finally.

Exposing the deployment

Now that our app is deployed, we need to expose it to the world. We can again use the imperative route to generate our declarative yaml:

kubectl expose deployment/hello-world --port 8080 --target-port 8080 --dry-run=client -o yaml > service.yaml

Our service.yaml looks like this:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: hello-world
  name: hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: hello-world
status:
  loadBalancer: {}

Let's apply it:

> kubectl apply -f service.yaml
service/hello-world created

> kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hello-world   ClusterIP   10.100.25.211   <none>        8080/TCP   2m5s

Our service is exposed with type ClusterIP, which means it is only accessible inside the cluster. We did not expose using a service type LoadBalancer. Does our Kubernetes cluster even support a service type of LoadBalancer? It is very IaaS specific. In public cloud providers such as EKS/AKS/GKE, you get it out of the box. In VMware Tanzu distributions you get several options to choose from. In a local Minikube or kind cluster, the answer is probably "nope". In other distros - YMMV.

At any case - we don't want to expose our application using an external IP address. We're not animals. We want a nice, beautiful URL.

For that, we need an to use an Ingress.

Adding an Ingress

In order to create ingresses in Kubernetes, we need an Ingress Controller .

Who installs that? There are quite a few choices available, such as NGINX, Contour, HAProxy and many others. The choice of ingress is probably a decision that should not be made solely by the development team.

In our case, let's install Contour, since it's fast and lightweight:

kubectl apply -f https://projectcontour.io/quickstart/contour.yaml

We now need to map our public or corporate DNS to point to the External IP that the ingress controller exposed. In case of Contour, we can find it with the following command:

> kubectl get svc -n projectcontour
NAMESPACE             NAME     TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)                      AGE
tanzu-system-ingress  envoy    LoadBalancer   10.100.101.15    ab915484edcec4e5ea4b532aa8a5340c-62fc50737cb73a64.elb.eu-west-2.amazonaws.com      80:30656/TCP,443:30358/TCP   2d4h

The External IP in my case is a host name since I'm running on AWS. For other providers, you'll probably get a real IP address.

We now need to tell our corporate domain or external domain registrar to map all traffic of certain addresses to the new ingress endpoint. We can have some fun and install an External DNS service in our cluster, which would even register the URLs for us, but for now we'll keep it simple. Let's map all *.apps.odedia.org to the hostname provided by AWS. I configured my domain registrar as follows:

Finally, we can create an ingress.yaml to describe our entry point. It's kind of a pain to use imperative commands with ingress so it's better to write the yaml ourselves. Contour also has a Custom Resource Definition called HTTPProxy which makes setting up routes easier, but we'll stick to the "standard" Ingress configuration for now:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
spec:
  rules:
  - host: hello.apps.odedia.org
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: hello-world
            port:
              number: 8080

We're telling the ingress that any request to hello.apps.odedia.org should be routed to the internal service hello-world that we created in the previous step.

Let's apply it:

> kubectl apply -f ingress.yaml 
ingress.networking.k8s.io/hello-world-ingress created

> kubectl get ingress    
NAME                  CLASS    HOSTS                   ADDRESS                                                                         PORTS   AGE
hello-world-ingress   <none>   hello.apps.odedia.org   ab915484edcec4e5ea4b532aa8a5340c-62fc50737cb73a64.elb.eu-west-2.amazonaws.com   80      8m59s

We finally have our application accessible through a web browser.

Although... I don't like this Not Secure part in the address bar. I would love to have it served as https://, since the greeting is top secret and should probably be encrypted in transit. I need to setup a TLS secret to make my application run as expected.

Creating free TLS certificates can be done via Let's Encrypt. Obtaining the certificate is beyond the scope of this article, but suffice to say that you can use anything from a TXT record in your domain registrar, to automating the process by installing cert-manager in your Kubernetes cluster.

Reality check

Remember that part in the beginning where we implemented our "Hello world" as a RestController? That was the only business logic we actually implemented so far. Everything else is just getting the plumbing to work.

Securing the ingress endpoint

Ok, so we have a signed certificate, somehow. We have our private key and public cert. Let's install it in our Kubernetes cluster:

> kubectl create secret tls hello-world-tls --cert=fullchain.pem --key=privkey.pem
secret/hello-world-tls created

We need to update our ingress with the TLS secret reference. I'm also adding an annotation to force SSL redirection:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    ingress.kubernetes.io/force-ssl-redirect: "true"
    kubernetes.io/ingress.class: contour
spec:
  tls:
  - secretName: hello-world-tls
    hosts:
    - 'hello.apps.odedia.org'
  rules:
  - host: hello.apps.odedia.org
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: hello-world
            port:
              number: 8080

Et voilà!

Just to be sure - we can see our connection is secure.

We're just getting started

We finally did it. We deployed our application to Kubernetes. We're done!

Well, not really... Turns out that developing new features is a never-ending task. We just got a new feature request to change the greeting to "Hello, everybody!". We now need to:

• Update our code
• Build, tag and push a new image
• Update and apply the deployment yaml

Is that all?

Of course not. There are still a lot of things we need to consider. To name just a few:

• How do new deployments get triggered? Will it be on every new push to the OCI registry?
• How will you reference the new image in your deployment? You need to have some sort of process to update the image reference with the new tag or its sha256 signature (please, don't use the latest tag ). Will you use a templating solution such as helm charts? Perhaps other solutions such as ytt and kapp from the carvel toolset?
• Come to think of it, what are we going to use for CI/CD? We have Jenkins, Azure Devops, Concourse, ArgoCD, Spinnaker, Tekton and many more. Each has its advantages and disadvantages. Tools like Jenkins and Concourse seem better at the CI part, since they are stateless and are not "Kubernetes aware". Tools like ArgoCD and Spinnaker make more sense for the CD part, since they are aware of what is currently in your clusters.
• What happens when you have a sudden spike in traffic? Can your deployment autoscale? We hardly considered it with our current, simple deployment. Will it only autoscale based on CPU/Memory using a Horizontal Pod Autoscaler, or will it be based on end-user experience and latency, using solutions such as Knative?
• Our application is ridiculously simple. A real app will probably at the very least connect to a database, a cache or a message broker. How do you intend to connect to such backing services? Who installs these backing services? How will you keep the credentials to those services safe?
• How will you perform blue-green deployments? What happens if a release has bugs, do you have a rollback strategy?
• How on earth are you going to debug this thing? Your applications should probably expose a debug port when testing against a local Kubernetes cluster during inner loop development, but disable it completely in other (pre-prod/production) scenarios.
• While on the subject of inner loop - how fast can you iterate on your code changes? If you'll have to do the whole docker build-tag-push + update deployment dance for each code change, you won't get much coding done. You may be looking at tools such as Skaffold , Tilt and/or Kustomize to improve iteration times, but you're still the one who has to write the configuration.
• What is your multi-cluster strategy? Moving from dev environment, to test, to pre-prod and eventually production - each is probably a separate Kubernetes environment, perhaps on different cloud providers. How will you make sure everything is setup correctly to accept the same application (Ingress controllers, TLS certificates, service accounts, Knative(?) etc.)
• How will you monitor the application? Will you use Prometheus and Grafana? Who's responsible to operate this service? Will you use a SaaS solution instead? How will you monitor application-specific KPIs such as HTTP Requests, logging levels, app-level environment variables etc.? These are not automatically exposed at a pod level.
• Here's a good one: Who handles security patching? Imagine you have 500 microservices running in production, and suddenly a new log4j-level exploit is released. How will you patch things as quickly and safely as possible? Moreover - how will you even know that your app is vulnerable?

There's just... too many stuff. Things that must be done, but are really just "glue" to get some code out the door.

Kubernetes needs a developer platform

• Developers should develop
• They should focus on writing code, and thinking about their code logic
• They care about connecting to third party services - their database, cache layer, message broker, legacy environment etc.
• They don’t really care about infrastructure, until things stop working
• They don’t really care about SSL certs, networking and all that jazz

Obviously, this is nothing new. Many companies and open source projects are trying to solve the developer crisis on Kubernetes. There are solutions from Weaveworks , Okteko , Ambassador , Digital Ocean, Red Hat and many more.

At Pivotal (now VMware Tanzu ), our approach is somewhat unique. As the creators of Spring, we love the idea of inversion of control . That principle is what allows Spring to autoconfigure itself to your needs without you having to tell it every single little tidbit. It's often referred to as the Hollywood principle: don't call us, we'll call you. Convention over configuration is a powerful feature, because most of the time, the default established conventions are exactly what you want. Wouldn't it be great to apply the same principals to a developer platform on Kubernetes?

The fact is, we already solved many of these challenges years ago by building a beloved, production-ready, enterprise-ready Platform-as-a-Service that is trusted by many Fortune 500 organizations. Pivotal Cloud Foundry (now Tanzu Application Service) allows developers to focus on their code, just cf push their application to an automated environment, easily connect to backing services, and enjoy automatic security patching whenever the platform operator upgrades the platform to a newer version. Back in the day, our haiku was:

Here’s my code

Run it on the cloud for me

I don’t care how

I compared the benefits of using Cloud Foundry instead of Kubernetes four years ago (that's exactly four years ago).

However, the landscape in our ever-changing industry has changed quite a bit during those four years:

• Kubernetes is the new IaaS. This was thought of as the future in 2017, but in 2022, it is just the present. Any container solution that doesn't use Kubernetes is losing points. Cloud Foundry is using BOSH and runs an (excellent) container runtime called Diego. Customers want the new IaaS.
• The reality is - sometimes, we do care how. Sometimes we want to dig deeper and make changes. If our company invested in a particular solution, we want to be able to integrate it easily into our developer platform. I'd say Cloud Foundry is 80% opinionated, 20% customizable, which is mind-blowing spectacular when you agree with those opinions. However, sometimes customers want a ratio that is skewed more towards the customizable part.
• In Kubernetes - everything is Infrastructure as Code. Some aspects in the Cloud Foundry ecosystem are imperative only, such as creating a backing service with cf create-service. This requires a bit of scripting to get a new Cloud Foundry org/space to the desired state. You also can't cf push a git repo - you can only push a local directory.
• IDE integration is good, but not spectacular. In order to debug a container running on the platform, developers needed to setup ssh tunneling using the cf ssh command.
• Setting up a local Cloud Foundry environment for a developer requires booting a resource-intensive virtual machine, and it lacks 1:1 parity with a full-blown Cloud Foundry installation.
• Working with multiple Cloud Foundry environments ("Foundations") is not a first-class feature. Developers basically need to login to env a, cf push, then login to env b, and cf push again. The full footprint of a Cloud Foundry installation is also rather resource-intensive, reaching around 10 virtual machines for a minimal installation.

Still, nothing can compete with the magic of giving just source code to a platform, and see an application served as a secured URL at the other side of that command. The result is basically what we did in this article and much more, with just a single command - cf push.

Looking forward

At Tanzu, we're building a developer platform for Kubernetes that tries to solve all these challenges and more. Tanzu Application Platform is our modern, cloud-native, Kubernetes-native Platform as a Service. Note: TAP is currently beta, not all features described below will be available at GA.

In the past year, I've had the privilege to contribute my thoughts (and rants...) to a group of very smart people inside Tanzu from R&D and the field, in order to make Tanzu Application Platform better and more aligned with real-world customer needs.

The idea is to take the same Inversion of Control concepts we love in Spring, and apply it to a developer platform for Kubernetes.

Also, if Kubernetes is indeed the new IaaS, we should be able to install the platform on any Kubernetes. This would remove the need for things like BOSH, for example. BOSH exists because no common multi-cloud infrastructure API existed when Cloud Foundry was built, so we had to build it ourselves. Now that we have such API with Kubernetes, TAP can technically run on any Kubernetes distro - EKS, GKE, AKS, Tanzu Kubernetes Grid, vSphere with Tanzu, OpenShift, Rancher, Tanzu Community Edition on AWS/Azure/Docker, Minikube, kind etc. (Disclaimer: not all Kubernetes distros are officially certified by VMware Tanzu).

TAP relies heavily on the notion of Custom Resource Definitions. This allows it to extend the Kubernetes API with additional features that would enable developers to interact with the platform at a higher level of abstraction. A platform operator only needs to install the platform once with a simple install command on a given Kubernetes cluster. The installation would take care of cross-cutting concerns such as configuring the Ingress, installing a container runtime such as Knative, setup automatic image building, scanning and signing, setup automatic build triggers on git commit, enable autoscaling of instances based on user activity, enable application monitoring and much more. Moreover, since we're using standard Kubernetes resources here, a platform operator can take almost any of the opinionated components in the platform and replace them with something else. The operator gets to choose how much he or she wants to move that slider of 80% opinionated, 20% customizable. If I were to create a Haiku for TAP, it would probably be something like:

Here’s my code

Run it on Kubernetes for me

Although my platform operator really cares how, I just want it to work

Ok, maybe I need to work on it some more 😄. But you get the idea. You get to choose the level of exposure to the underpinnings of the platform as it fits your needs.

Here's a good example for this: TAP uses Knative as its runtime. You may decide you couldn't care less and just use the equivalent of cf push in TAP (which is tanzu apps workload create, by the way), but you may also decide you do care, and use the kn Knative cli to see what goes on with the Knative revisions under the hood. You can always just use kubectl to see the primitives themselves (Pods, ReplicaSets etc.), and TAP has a very nice UI to see all runtime resources. Also, if for some reason your organization chooses to avoid Knative, your platform operator may choose something else, such as a deployment yaml like we used in this article.

Here are some of the capabilities that TAP solves as a development platform for Kubernetes:

• Developers start with production-ready templates for their source code. An architect or project manager can expose to his team templates provided by Tanzu, or create his own, based on specific corporate requirements. You can also choose just how much customizations you want to expose to the developers. This removes a lot of overhead for "getting started", and also applies company-wide policies for what makes a good starting point for development.

• Go from source to URL in 90 seconds. That's basically everything we implemented in this article. We took all the learnings from Cloud Foundry and implemented it in a Kubernetes-native paradigm. Cloud Native Buildpacks are still the secret sauce of open source that builds our production-ready images.
• IDE integration. You can continue development against a live container running in a local or a remote Kubernetes cluster, including live updates based on any code change, and even remote debugging.

• Application-specific monitoring - HTTP requests, env variables, deployed configuration and Beans (in Spring), even change the logging level of a particular class at runtime if you want to.

• Multi-cluster support. Take an artifact from source to production, while moving the deployable artifact across different clusters.
• Image security scanning, patching and signing. Cloud Native Buildpacks will patch the OS level and the middleware. Scanning and signing will also alert you on CVEs in your own code or dependencies, and optionally block deployment to the cluster if the CVE is critical.
• The ability to perform service bindings to backing services - declare a desired access to a service such as RabbitMQ based on its name, and the platform will bind the service to your application. For some popular services, it will even auto-configure Spring to connect to the service with zero configuration. This makes your app 100% portable since no IPs or passwords are hanging around your source code (and potentially end up in your git repo by mistake).
• Service binding can also be cross-cluster. Setup a services cluster for your message brokers and databases, and a workload cluster for your apps.
• UI dashboards that display information about your apps, their runtime state, and even allow you to write wiki-style documentation for your microservices.

• Access to workshops developed in-house or by third parties, using a web-based workshop environment that gives every student access to a Kubernetes namespace and even web-based VSCode (Just bring a Chrome browser and an Internet connection).

I plan to create several getting started guides, tutorials and videos around TAP in the coming months, so be sure to subscribe for updates if you're interested.

Conclusion

Kubernetes was not my first choice as a developer. I always felt it is too low level for me to get things done. It seems that even the co-founder of Kubernetes agrees with this. At Spring One 2021, Craig McLuckie said that they never meant for developers to interact directly with the wall of YAML:

That is why I love Cloud Foundry so much. It allows me to focus on my code, and have the infrastructure get out of the way.

But there is a lot to be said on the benefits of using a ubiquitous infrastructure API. The benefits are massive. Vendor lock-in is basically non existent. Just use whatever fits your needs and your budget.

Creating a platform that is focused on developer productivity on Kubernetes lets you enjoy the best of both worlds. Developers can keep focusing on their code, platform operators can choose the best tech stack that matches their requirements, IT can operate low-level Kubernetes, and the security team can be alerted whenever a security issue may arise (and in many cases, automatically patch it).

Epilogue

So, how would I get the same results (and much more - for example automatic autoscaling based on HTTP traffic) as our example above, using Tanzu Application Platform? Here it is (The yaml is auto-generated and it not a part of the command):

> tanzu apps workload create tanzu-java-web-app \
--git-repo https://github.com/odedia/spring-on-k8s \
--git-branch main --type web --yes

Create workload:
      1 + |---
      2 + |apiVersion: carto.run/v1alpha1
      3 + |kind: Workload
      4 + |metadata:
      5 + |  labels:
      6 + |    apps.tanzu.vmware.com/workload-type: web
      7 + |  name: tanzu-java-web-app
      8 + |  namespace: default
      9 + |spec:
     10 + |  source:
     11 + |    git:
     12 + |      ref:
     13 + |        branch: main
     14 + |      url: https://github.com/odedia/spring-on-k8s 

Created workload "tanzu-java-web-app"

To get the URL, just ask for it:

> tanzu apps workload get tanzu-java-web-app     
# tanzu-java-web-app: Ready
---
lastTransitionTime: "2022-01-01T01:56:14Z"
message: ""
reason: Ready
status: "True"
type: Ready

Workload pods
NAME                                                   STATE       AGE
tanzu-java-web-app-00002-deployment-6f4cd7cb8c-w9shd   Running     66s
tanzu-java-web-app-build-1-build-pod                   Succeeded   2m16s
tanzu-java-web-app-config-writer-6wllp-pod             Succeeded   92s

Workload Knative Services
NAME                 READY     URL
tanzu-java-web-app   Ready   http://tanzu-java-web-app-default.apps.odedia.org

For more information on Tanzu Application Platform, check out the download page and the docs.

Special thanks to Thomas Risberg, Timo Salm and Lekha Pillai for providing feedback on this article

I work for VMware Tanzu by Broadcom as a principal solutions architect and advocate for Spring, focused on cloud native platform such as Cloud Foundry and Tanzu Platform for Kubernetes.

I helped design large-scale, cloud-native solutions running on Cloud Foundry and Kubernetes, using Spring Boot and Spring Cloud.

I’m highly skilled in Kubernetes, Java, Spring Boot, Spring Cloud, Apache Kafka, Event Sourcing and DevOps practices.

I enjoy photography, video editing, and the music of the King of Pop.

Here is a version of this video in Hebrew:

Our special guest was Spring Boot and Spring Cloud creator Dave Syer. He discussed Spring Cloud Function and Serverless spring.

Dr. Syer's talk is available here:

If you want get a deeper understanding of how the Pivotal Cloud Foundry installation works, including the use of BOSH, Ops Manager and the networking you need to setup, have a look at PCF on Azure - the Hard Way.

Go to the Pivotal Cloud Foundry on Microsoft Azure page and click Get it now followed by Continue and then Create:

p>

You need to provide 5 parameters for the installation:

Run the following command on a linux machine:

mkdir ~/certs
ssh-keygen -t rsa -b 2048 -f ~/certs/pcf-on-azure -P ""

Copy the contents of ~/certs/pcf-on-azure.pub to the field SSH public key. Keep the file ~/certs/pcf-on-azure in a secured place for future reference.

You will need a Service Principal JSON file for the Service Principal field. The fields in the JSON are as follows:

{
  "subscriptionID": "SUBSCRIPTION-ID",
  "tenantID": "TENANT-ID",
  "clientID": "SERVICE-PRINCIPAL-NAME",
  "clientSecret": "SERVICE-PRINCIPAL-PASSWORD"
}

If you don't have it yet, install the Azure CLI from https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest

Login with the az cli:

az login

Confirm the login in the pop-up web page. Once done you will see a list fo accounts in the command prompt.

Review the list of accounts by running:

az account list

Important: If you have multiple accounts, make sure you set the default to the account you want to use for the installation! The default account is marked as isDefault . You can change it using:

az account set -s <id-field-from-response-json>

You now have two out of the four required fields:

• The id from az account list is the subscriptionID. Don't ask me why, I'm sure Microsoft had a good reason :)
• The tenantId from az account list is the tenantID field. Make sure that in the latter it is ID, not Id.

We now need to create a service principal for the installation.

First, save some environment variables for later use:

CLIENT_SECRET=<choose a secure secret>
USER_ID=<choose a application user id, for example Oded>
SUBSCRIPTION_ID=<id field from response json above while running az account list>
TENANT_ID=<tenantId field from response json above while running az account list>

Important: Choose a CLIENT_SECRET that is at least 40 characters long and contains alphanumeric characters, digits and at least one underscore. Otherwise the deployment will later fail on password validation.

Now, create an active directory application:

az ad app create \
--display-name "Service Principal for BOSH" \
--password $CLIENT_SECRET \
--homepage "http://BOSHAzureCPI" \
--identifier-uris "http://${USER_ID}BOSHAzureCPI"

(Tip: if you ever need to delete the AAD use this: az ad app delete --id "http://${USER_ID}BOSHAzureCPI")

Create a service principal from the AAD:

az ad sp create --id `az ad app show --id http://${USER_ID}BOSHAzureCPI | jq -r .appId`

Wait for about 30 seconds for Azure active directory updates to progapate, then add the contributor role:

az role assignment create --assignee http://${USER_ID}BOSHAzureCPI --role "Contributor" --scope /subscriptions/${SUBSCRIPTION_ID}

Login with the Application user:

az login \
--username `az ad app show --id http://${USER_ID}BOSHAzureCPI | jq -r .appId` \
--password $CLIENT_SECRET \
--service-principal \
--tenant ${TENANT_ID}

Enable compute, netowrk and storage access:

az provider register --namespace Microsoft.Storage
az provider register --namespace Microsoft.Network
az provider register --namespace Microsoft.Compute

Logout of the application user and login again to your regular azure account:

az logout
az login

You now have the two remaining fields:

• The ClientID can be extracted by running az ad app show --id http://${USER_ID}BOSHAzureCPI | jq -r .appId
• The ClientSecret is the value of $CLIENT_SECRET you set above.

A final JSON might look like this:

{
  "subscriptionID": "24A7CC81-402C-44EB-8DFE-96A8DB670E37",
  "tenantID": "BF7500C3-A3D4-411F-AA5B-606CBE903FC4",
  "clientID": "23536697-C5F6-49DD-9E2F-68D36FFDCA22",
  "clientSecret": "My-Super-Secret-Secret"
}

Save this file as azure-creds.json, then upload it to the web page under the Service Principal field.

Under "Resource group", choose "Create new" and name the resource group "pcf".

Choose a Location you want to deploy to.

Until Pivotal Network Token, enter your Pivotal Network token from Pivnet at network.pivotal.io. You can find (once logged in) under your User Name --> Edit Profiles --> Request New Refresh Token. It's best to keep this token in a secured place like LastPass and not recreate it every time you need to.

Before continuing, make sure you accepted the EULAs for the links below at Pivnet. In each of the following links, you will be prompted to accept an End-User License Agreement. Once you accepted the EULA, a download will begin, but you can immediatly cancel it. You account is now linked to accepting the EULA and so the installation from Azure can download the files.

https://network.pivotal.io/products/ops-manager/ (Choose latest 2.4 version and then download Pivotal Cloud Foundry Ops Manager YAML for Azure. Do the same for the latest 2.5 version) https://network.pivotal.io/products/elastic-runtime/ (Choose latest 2.4 version and then download Small Footprint PAS. Do the same for the latest 2.5) https://network.pivotal.io/products/azure-service-broker/ (Choose latest version and then download Microsoft Azure Service Broker)

You now have all required data for installation:

Click "OK". In the next page, wait for the validation to complete and click "OK" again. In the last screen, click "Create".

Your deployment will now begin. You can track the progress under Resource Groups --> pcf --> Deployments. You will receive a notification in the Azure portal when the Deployment was successful, however note that there are additional installations done after the deployment is complete.

When the deployment is done, click on the deployment pivotal.pivotal-cloud-foundry-.... and then Outputs. You will see the login URL and password for Ops Manager:

Go to the URL listed under opsMan-FQDN. The username is admin and the password is located under Outputs-->password.

You can now track the progress of the installation in Ops Manager.

When the BOSH Direcor installation completes, it might take up to 1/2 an hour for the PAS installation to continue. You can track the progress under Resource Group --> Activity Log.

Still, developers do require access to some aspects of the Kubernetes lifecycle, mainly the Kubernetes dashboard. However, that dashboard requires an access token that is found in the KUBECONFIG file.

In PKS, it is possible to get the KUBECONFIG file that includes the access token by using the pks cli, but if kubectl is off limits, the pks cli is definetly off limits.

The solution? Run a script that talks to the UAA backend to get the access token. Pivotal and VMware provide a script under this community support article.

However, even that can be a challenge for many enterprises. Windows is still king, and running shell scripts is not possible, let alone installation of additional utilities such as jq.

The solution? Get the access token from a Jenkins job that developers already have access to in their daily work.

Let's create a new Jenkins Freestyle job:

Check the box to Discard old builds and set Max # of builds to keep to 0. You don't want to keep a history of the access tokens.

Check This project is parameterized and add the following parameters:

Make sure that PKS_PASSWORD_RAW is set to type Password Parameter.

Under Build choose Execute shell:

Paste the following script. It has been modified from the original so it would not require jq:

#!/bin/bash -e
# v 0.0.5.1

# get-pks-k8s-config.sh
# gmerlin@vmware.com
# adapted for Jenkins by Oded Shopen (odedia.org)

urlencode() {
    local l=${#1}
    for (( i = 0 ; i < l ; i++ )); do
        local c=${1:i:1}
        case "$c" in
            [a-zA-Z0-9.~_-]) printf "$c" ;;
            ' ') printf + ;;
            *) printf '%%%.2X' "'$c"
        esac
    done
}


PKS_PASSWORD=$(urlencode $PKS_PASSWORD_RAW)

# Collect Tokens from UAA
CURL_CMD="curl 'https://${PKS_API}:8443/oauth/token' -sk -X POST -H 'Accept: application/json' -d \"client_id=pks_cluster_client&client_secret=\"\"&grant_type=password&username=${PKS_USER}&password=\"${PKS_PASSWORD}\"&response_type=id_token\""

TOKENS=$(eval $CURL_CMD | python -c "import sys, json; print json.load(sys.stdin)['id_token']" ) 
echo -e "\n"
echo $TOKENS
echo -e  "\n"

Save the job.

While running the job, provide all required parameters:

The output will present your access token:

Good luck!

Still, being Kubernetes, it would be nice to treat an SMB share as yet another Persistent Volume and abstract away the protocol being used.

I've found a few solutions for this online, but the only one that seemed valuable and from a well known vendor was the FlexVolume for Azure by Microsoft. This project exposes several Azure services as PersistentVolumes to Kubernetes.

The drawback of this project is that it was developed specifically for Azure and AKS (Azure Kubernetes Service).

The biggest issue was that the installation required you to install specific binaries on the worker nodes, specifically jq. While that may be a feasible outcome on AKS, it was not a realistic approach for PKS. In PKS, BOSH manages the Worker nodes, patches them, upgrades then or just repaves (i.e. resets) them to a known state. Therefore, any "snowflake" binary installation on the working nodes would disappear on the next upgrade cycle.

I've been working with my collegue Stuart Charlton (@svrc) to come up with a solution, and we came up with this fork.

It differs from the upstream Azure project in the following ways:

1. hostPath settings are set to match their expected location on PKS.
2. jq is being installed as part of the DameonSet deployment to Kubernetes, so no prior installation is required. Upon upgrade/resize of the cluster, each new/upgraded node would run the DameonSet again and install jq again. Microsoft states that cifs-utils is another required binary but it is included in the PKS distribution so it already exists on the worker nodes.

I tested this on a PKS 1.3 cluster. I was able to resize, repave and upgrade the cluster, and the SMB persistent volume was still there waiting for me.

Prerequisites

You will need a Kubernetes clusters where Privileged Containers are enabled. Under your Plan settings in Ops Manager, check the box for Enable Privileged Containers - Use with caution and Disable DenyEscalatingExec:

You will need an Egress network policy in your Kubernetes Cluster to allow outbound traffic on port 445 (SMB). If you're running PKS on-premise with NSX-T, it's as easy as applying a Network Policy yaml. For example:

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
    name: smb-network-policy
    namespace: default
spec:
    podSelector:
      matchLabels:
        app: nginx-flex-smb
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: <your target IP Block>/32
    ports:
    - protocol: TCP
      port: 445

Persistent Volumes also require a default storage class. If you're running PKS on vSphere, you can create the default storage class with the yaml below:

  kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
    name: thin
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
  provisioner: kubernetes.io/vsphere-volume
  parameters:
      diskformat: thin

Apply the Storage Class:

kubectl apply -f storage-class-vsphere.yml

How to Install

Create a DaemonSet with the FlexVolume driver. A DaemonSet is a pod that runs on each one of the worker nodes in Kubernetes.

kubectl apply -f https://raw.githubusercontent.com/odedia/kubernetes-volume-drivers/master/flexvolume/smb/deployment/smb-flexvol-installer.yaml

Check DaemonSet status:

watch kubectl describe daemonset smb-flexvol-installer --namespace=kube-system
watch kubectl get po --namespace=kube-system -o wide

Once you see your pods in a Running state, you can continue.

Create a Kubernetes secret to hold your SMB credentials:

kubectl create secret generic smbcreds --from-literal username=USERNAME --from-literal password="PASSWORD" --type="microsoft.com/smb"

Replace USERNAME and PASSWORD with your credentials.

Running a pod with direct mount

You can define the SMB access on the pod definition itself, or via a persistent volume claim. For direct configuration, you will need to define the volumes section of your Pod definition. Here's a sample Pod you can use for testing:

https://raw.githubusercontent.com/odedia/kubernetes-volume-drivers/master/flexvolume/smb/nginx-flex-smb.yaml

Edit the yaml contents and replace the source line with the address of your SMB server and shared folder.

Once done, apply the yaml:

kubectl apply -f nginx-flex-smb.yaml

Running a pod with mount via Persistent Volume Claim

You can setup the SMB share via a Persistent Volume Claim (PVC).

Download the Persistent Volume (PV) yaml:

wget https://raw.githubusercontent.com/odedia/kubernetes-volume-drivers/master/flexvolume/smb/pv-smb-flexvol.yaml

Edit the yaml contents and replace the source line with the address of your SMB server and shared folder. Apply the changes:

kubectl apply -f pv-smb-flexvol.yaml

Create a PVC that is bound to the newly created persistent volume:

 kubectl apply -f https://raw.githubusercontent.com/odedia/kubernetes-volume-drivers/master/flexvolume/smb/pvc-smb-flexvol.yaml

Check the status of the PV and the PVC:

kubectl get pv
kubectl get pvc

Run a test pod with this PVC:

 kubectl apply -f https://raw.githubusercontent.com/odedia/kubernetes-volume-drivers/master/flexvolume/smb/nginx-flex-smb-pvc.yaml

Verify the volume mount

SSH into the pod:

kubectl exec -it nginx-flex-smb -- bash

Run df -h. You should see your SMB share mounted under /data:

root@nginx-flex-smb:/# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
overlay                                    291G  3.2G  288G   2% /
tmpfs                                      3.4G     0  3.4G   0% /dev
tmpfs                                      3.4G     0  3.4G   0% /sys/fs/cgroup
//myfileserver.com/my-shared-folder           25G   64K   25G   1% /data
/dev/sda1                                  291G  3.2G  288G   2% /etc/hosts
shm                                         64M     0   64M   0% /dev/shm
tmpfs                                      3.4G   12K  3.4G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                      3.4G     0  3.4G   0% /sys/firmware

Credits: This tutorial is largly based on the excellent work by Alan McGinlay on github, my small contributions were:

• One terraform for both PAS and PKS since it’s now supported on the same ops manager.
• Added terraform entries for VMware harbor (it’s not included in the official PKS terraform for some reason)
• Let’s encrypt certificates for concourse and PCF.
• Separation of main pipeline and “fetch artifacts”
• Additional marketplace services
• Some minor fixes and customizations such as triggers of upload-and-configure when new artifacts are downloaded.

The following steps use Control Tower to build a Concourse instance on Google Cloud Platform, then uses a combination of GCS buckets, Credhub, a suite of Platform Automation tools and a single Concourse pipeline to deploy (and upgrade) the entire OpsMan and PCF product stack directly from the Pivotal Network.

This fork was adapted to use a terraform script that paves both Pivotal Application Service and Pivotal Container Service, and an updated pipeline that runs both products on the same Ops Manager.

Setup proper quotas

You will need quite a bit of resources for the entire installation of both Pivotal Application Service, Pivotal Container Service and all the included marketplace services in this installation. You can remove services you don't need from the final pipeline below (under ops-manager-automation-cc/ci/pivotal-cloud-foundry/pipeline.yml). The following quotas are required for the full installation in a given region:

Setup DNS namespaces

You need to setup two NS records at your domain registrar for this installation. One will be for concourse and one would be for the platform. You want to seperate the two into different subdomains so there will not be any dependency between them. Create a new NS record for concourse, for example devops, and give it an NS. Give a different name for the platform, for example cf. Here's an example of setting it on Google domains:

You need to provide all the nameserver records from Google Cloud for each of the NS records. The list is:

ns-cloud-a1.googledomains.com.
ns-cloud-b1.googledomains.com.
ns-cloud-c1.googledomains.com.
ns-cloud-d1.googledomains.com.
ns-cloud-e1.googledomains.com.
ns-cloud-a2.googledomains.com.
ns-cloud-b2.googledomains.com.
ns-cloud-c2.googledomains.com.
ns-cloud-d2.googledomains.com.
ns-cloud-e2.googledomains.com.
ns-cloud-a3.googledomains.com.
ns-cloud-b3.googledomains.com.
ns-cloud-c3.googledomains.com.
ns-cloud-d3.googledomains.com.
ns-cloud-e3.googledomains.com.
ns-cloud-a4.googledomains.com.
ns-cloud-b4.googledomains.com.
ns-cloud-c4.googledomains.com.
ns-cloud-d4.googledomains.com.
ns-cloud-e4.googledomains.com.

Fork this repository

I recommend forking this repository so you can:

• Make modifications to suit your own requirements
• Protect your active pipelines from config changes made here

Recycling GCP projects

If you wish to re-use an existing GCP project for this exercise, it is often useful to clean up any existing resources beforehand. For guidance, follow these instructions.

Create your jumpbox from your local machine or Google Cloud Shell

GCP_PROJECT_ID=<TARGET_GCP_PROJECT_ID>
GCP_REGION=<TARGET_REGION>
gcloud auth login --project ${GCP_PROJECT_ID} --quiet # ... if necessary

gcloud services enable compute.googleapis.com \
  --project "${GCP_PROJECT_ID}"

gcloud compute instances create "jbox-cc" \
  --image-project "ubuntu-os-cloud" \
  --image-family "ubuntu-1804-lts" \
  --boot-disk-size "200" \
  --machine-type=g1-small \
  --project "${GCP_PROJECT_ID}" \
  --zone "${GCP_REGION}"-b

Move to the jumpbox and log in to GCP

gcloud compute ssh ubuntu@jbox-cc \
  --project "${GCP_PROJECT_ID}" \
  --zone "${GCP_REGION}"-b

gcloud auth login --quiet

All following commands should be executed from the jumpbox unless otherwsie instructed.

Prepare your environment file

cat > ~/.env << EOF
# *** your environment-specific variables will go here ***
PIVNET_UAA_REFRESH_TOKEN=CHANGE_ME_PIVNET_UAA_REFRESH_TOKEN  # e.g. xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-r
PCF_DOMAIN_NAME=CHANGE_ME_DOMAIN_NAME                        # e.g. "mydomain.com", "pal.pivotal.io", "pivotaledu.io", etc.
PCF_SUBDOMAIN_NAME=CHANGE_ME_SUBDOMAIN_NAME                  # e.g. "mypks", "mypas", "cls66env99", "maroon", etc.
GITHUB_PUBLIC_REPO=CHANGE_ME_GITHUB_PUBLIC_REPO              # e.g. https://github.com/odedia/pivotal-platform-automation-gcp.git
GCP_REGION=CHANGE_ME_GCP_REGION                              # e.g. europe-west2, us-central1 etc.
CONCOURSE_SUBDOMAIN_NAME=CHANGE_ME_CONCOURSE_SUBDOMAIN       #e.g. "devops"
GCP_AZ1=CHANGE_ME_GCP_AZ1                              # e.g. europe-west2-a, us-central1-a etc.
GCP_AZ2=CHANGE_ME_GCP_AZ2                              # e.g. europe-west2-b, us-central1-b etc.
GCP_AZ3=CHANGE_ME_GCP_AZ3                              # e.g. europe-west2-c, us-central1-c etc.
PRODUCT_SLUG=pivotal-cloud-foundry
MY_EMAIL=CHANGE_ME_EMAIL_ADDRESS                      #email address for Lets Encrypt

export OM_TARGET=https://opsman.\${PCF_SUBDOMAIN_NAME}.\${PCF_DOMAIN_NAME}
export OM_USERNAME=admin
export OM_PASSWORD=$(uuidgen)
export RABBITMQ_MULTITENANT_ADMIN_PASSWORD=$(uuidgen)
export OM_DECRYPTION_PASSPHRASE=\${OM_PASSWORD}
export OM_SKIP_SSL_VALIDATION=true
EOF

Before continuing, open the .env file and update the CHANGE_ME values accordingly.

Ensure these variables get set into the shell every time the ubuntu user connects to the jumpbox:

echo "source ~/.env" >> ~/.bashrc

Load the variables into your shell with the source command so we can use them immediately:

source ~/.env

Prepare jumpbox and generate service account

gcloud services enable iam.googleapis.com --async
gcloud services enable cloudresourcemanager.googleapis.com --async
gcloud services enable dns.googleapis.com --async
gcloud services enable sqladmin.googleapis.com --async

sudo apt update --yes && \
sudo apt install --yes jq && \
sudo apt install --yes build-essential && \
sudo apt install --yes ruby-dev && \
sudo apt-get install software-properties-common --yes && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -  && \
sudo apt-get update --yes && \
sudo apt-get install software-properties-common  --yes && \
sudo add-apt-repository ppa:certbot/certbot --yes && \
sudo apt-get update --yes && \
sudo apt-get install certbot --yes && \
sudo apt-get install python-minimal --yes && \
sudo apt-get install python-pip python-dev build-essential --yes && \
sudo apt-get install python-setuptools --yes && \
sudo pip install cryptography && \
sudo pip install certbot-dns-google

cd ~

FLY_VERSION=5.6.0
wget -O fly.tgz https://github.com/concourse/concourse/releases/download/v${FLY_VERSION}/fly-${FLY_VERSION}-linux-amd64.tgz && \
  tar -xvf fly.tgz && \
  sudo mv fly /usr/local/bin && \
  rm fly.tgz

CT_VERSION=0.8.3
wget -O control-tower https://github.com/EngineerBetter/control-tower/releases/download/${CT_VERSION}/control-tower-linux-amd64 && \
  chmod +x control-tower && \
  sudo mv control-tower /usr/local/bin/

OM_VERSION=0.51.0
wget -O om https://github.com/pivotal-cf/om/releases/download/${OM_VERSION}/om-linux && \
  chmod +x om && \
  sudo mv om /usr/local/bin/

PN_VERSION=0.0.55
wget -O pivnet https://github.com/pivotal-cf/pivnet-cli/releases/download/v${PN_VERSION}/pivnet-linux-amd64-${PN_VERSION} && \
  chmod +x pivnet && \
  sudo mv pivnet /usr/local/bin/

BOSH_VERSION=5.4.0
wget -O bosh https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-${BOSH_VERSION}-linux-amd64 && \
  chmod +x bosh && \
  sudo mv bosh /usr/local/bin/

CHUB_VERSION=2.6.1
wget -O credhub.tgz https://github.com/cloudfoundry-incubator/credhub-cli/releases/download/${CHUB_VERSION}/credhub-linux-${CHUB_VERSION}.tgz && \
  tar -xvf credhub.tgz && \
  sudo mv credhub /usr/local/bin && \
  rm credhub.tgz

BBL_VERSION=7.6.0
wget -O bbl https://github.com/cloudfoundry/bosh-bootloader/releases/download/v7.6.0/bbl-v${BBL_VERSION}_linux_x86-64 && \
  chmod +x bbl && \
  sudo mv bbl /usr/local/bin

TF_VERSION=0.11.13
wget -O terraform.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip && \
  unzip terraform.zip && \
  sudo mv terraform /usr/local/bin && \
  rm terraform.zip

git clone https://github.com/odedia/terraforming-gcp && \
  mv terraforming-gcp terraforming

gcloud iam service-accounts create p-service --display-name "Pivotal Service Account"

gcloud projects add-iam-policy-binding $(gcloud config get-value core/project) \
  --member "serviceAccount:p-service@$(gcloud config get-value core/project).iam.gserviceaccount.com" \
  --role 'roles/owner'

cd ~
gcloud iam service-accounts keys create 'gcp_credentials.json' \
  --iam-account "p-service@$(gcloud config get-value core/project).iam.gserviceaccount.com"

Clone this repo

The scripts, pipelines and config you need to complete the following steps are inside this repo, so clone it to your jumpbox:

git clone ${GITHUB_PUBLIC_REPO} ~/ops-manager-automation-cc

Create a certificate signed by Let's Encrypt

Create a Google Cloud Zone DNS entry:

Run the following script to create a certificate and key for the installation:

gcloud dns managed-zones create ${PCF_SUBDOMAIN_NAME}-zone --dns-name ${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}. --description "PCF ZONE"
DOMAIN=${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME} EMAIL=${MY_EMAIL} ~/ops-manager-automation-cc/bin/certbot.sh
gcloud dns managed-zones delete ${PCF_SUBDOMAIN_NAME}-zone

Note: If you face issues with AttributeError: 'module' object has no attribute 'Locale', please remove parsedatetime 2.6 by running pip uninstall parsedatetime and then run pip install parsedatetime==2.5.

Configure Terraform

cat > ~/terraform.tfvars <<-EOF
dns_suffix             = "${PCF_DOMAIN_NAME}"
env_name               = "${PCF_SUBDOMAIN_NAME}"
region                 = "${GCP_REGION}"
zones                  = ["${GCP_AZ1}", "${GCP_AZ2}", "${GCP_AZ3}"]
project                = "$(gcloud config get-value core/project)"
opsman_image_url       = ""
opsman_vm              = 0
create_gcs_buckets     = "false"
external_database      = 0
isolation_segment      = 0
ssl_cert            = <<SSL_CERT
$(sudo cat /etc/letsencrypt/live/${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}/fullchain.pem)
SSL_CERT
ssl_private_key     = <<SSL_KEY
$(sudo cat /etc/letsencrypt/live/${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}/privkey.pem)
SSL_KEY
service_account_key = <<SERVICE_ACCOUNT_KEY
$(cat ~/gcp_credentials.json)
SERVICE_ACCOUNT_KEY
EOF

Note the opsman_image_url == "" setting which prohibits Terraform from downloading and deploying the Ops Manager VM. The Concourse pipelines will take responsibility for this.

Terraform the infrastructure

The PKS and PAS platforms have different baseline infrastructure requirements which are configured from separate dedicated directories. Terraform is directory-sensitive and needs local access to your customized terraform.tfvars files so symlink it in from the home directory.

cd ~/terraforming/terraforming-pcf
ln -s ~/terraform.tfvars .

Launch Terraform

Confirm you're in the correct directory for your chosen platform and terraform.tfvars is present, then execute the following:

terraform init
terraform apply --auto-approve

This will take about 2 mins to complete.

Install Concourse

We use Control Tower to install Concourse, as follows:

gcloud dns managed-zones create ${CONCOURSE_SUBDOMAIN_NAME}-zone --dns-name ${CONCOURSE_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}. --description "Devops zone"
GOOGLE_APPLICATION_CREDENTIALS=~/gcp_credentials.json \
  control-tower deploy \
    --region ${GCP_REGION} \
    --iaas gcp \
    --workers 2 \
    --domain concourse.${CONCOURSE_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME} \
    pautomation

This will take about 20 mins to complete.

Persist a few credentials

INFO=$(GOOGLE_APPLICATION_CREDENTIALS=~/gcp_credentials.json \
  control-tower info \
    --region ${GCP_REGION} \
    --iaas gcp \
    --json \
    pautomation
)

echo "CC_ADMIN_PASSWD=$(echo ${INFO} | jq --raw-output .config.concourse_password)" >> ~/.env
echo "CREDHUB_CA_CERT='$(echo ${INFO} | jq --raw-output .config.credhub_ca_cert)'" >> ~/.env
echo "CREDHUB_CLIENT=credhub_admin" >> ~/.env
echo "CREDHUB_SECRET=$(echo ${INFO} | jq --raw-output .config.credhub_admin_client_secret)" >> ~/.env
echo "CREDHUB_SERVER=$(echo ${INFO} | jq --raw-output .config.credhub_url)" >> ~/.env
echo 'eval "$(GOOGLE_APPLICATION_CREDENTIALS=~/gcp_credentials.json \
  control-tower info \
    --region ${GCP_REGION} \
    --iaas gcp \
    --env pautomation)"' >> ~/.env

source ~/.env

Verify BOSH and Credhub connectivity

bosh env
credhub --version

Check Concourse targets and check the pre-configured pipeline:

fly targets
fly -t control-tower-pautomation pipelines

Navigate to the url shown for fly targets.

Use admin user and the value of CC_ADMIN_PASSWD to login and see the pre-configured pipeline.

Note control-tower will log you in but valid access tokens will expire every 24 hours. The command to log back in is:

fly -t control-tower-pautomation login --username admin --password ${CC_ADMIN_PASSWD}

Set up dedicated GCS bucket for downloads

gsutil mb -c regional -l ${GCP_REGION} gs://${PCF_SUBDOMAIN_NAME}-concourse-resources
gsutil versioning set on gs://${PCF_SUBDOMAIN_NAME}-concourse-resources

Add a dummy state file

The state.yml file is produced by the create-vm platform automation task and serves as a flag to indicate that an Ops Manager exists. We currently store the state.yml file in GCS. The install-opsman job also consumes this file so it can short-circuit the create-vm task if an Ops Manager does exist. This is a mandatory input and does not exist by default so we create a dummy state.yml file to kick off proceedings. Storing the state.yml file in git may work around this edge case but, arguably, GCS/S3 is a more appropriate home.

echo "---" > ~/state.yml
gsutil cp ~/state.yml gs://${PCF_SUBDOMAIN_NAME}-concourse-resources/

If required, be aware that versioned buckets require you to use gsutil rm -a to take files fully out of view.

Store secrets in Credhub

credhub set -n pivnet-api-token -t value -v "${PIVNET_UAA_REFRESH_TOKEN}"
credhub set -n domain-name -t value -v "${PCF_DOMAIN_NAME}"
credhub set -n subdomain-name -t value -v "${PCF_SUBDOMAIN_NAME}"
credhub set -n gcp-project-id -t value -v "$(gcloud config get-value core/project)"
credhub set -n opsman-public-ip -t value -v "$(dig +short opsman.${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME})"
credhub set -n gcp-credentials -t value -v "$(cat ~/gcp_credentials.json)"
credhub set -n om-target -t value -v "${OM_TARGET}"
credhub set -n om-skip-ssl-validation -t value -v "${OM_SKIP_SSL_VALIDATION}"
credhub set -n om-username -t value -v "${OM_USERNAME}"
credhub set -n om-password -t value -v "${OM_PASSWORD}"
credhub set -n om-decryption-passphrase -t value -v "${OM_DECRYPTION_PASSPHRASE}"
credhub set -n domain-crt -t value -v "$(sudo cat /etc/letsencrypt/live/${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}/fullchain.pem)"
credhub set -n domain-key -t value -v "$(sudo cat /etc/letsencrypt/live/${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}/privkey.pem)"
credhub set -n region -t value -v "${GCP_REGION}"
credhub set -n az1 -t value -v "${GCP_AZ1}"
credhub set -n az2 -t value -v "${GCP_AZ2}"
credhub set -n az3 -t value -v "${GCP_AZ3}"
credhub set -n rabbitmq-multitenant-password -t value -v "${RABBITMQ_MULTITENANT_ADMIN_PASSWORD}"

Take a moment to review these settings with credhub get -n <NAME>.

Build the pipeline

Create a private.yml to contain the secrets required by pipeline.yml:

cat > ~/private.yml << EOF
---
product-slug: ${PRODUCT_SLUG}
config-uri: ${GITHUB_PUBLIC_REPO}
gcp-credentials: |
$(cat ~/gcp_credentials.json | sed 's/^/  /')
gcs-bucket: ${PCF_SUBDOMAIN_NAME}-concourse-resources
pivnet-token: ${PIVNET_UAA_REFRESH_TOKEN}
credhub-ca-cert: |
$(echo $CREDHUB_CA_CERT | sed 's/- /-\n/g; s/ -/\n-/g' | sed '/CERTIFICATE/! s/ /\n/g' | sed 's/^/  /')
credhub-client: ${CREDHUB_CLIENT}
credhub-secret: ${CREDHUB_SECRET}
credhub-server: ${CREDHUB_SERVER}
EOF

Set and unpause the pipelines:

Create the "fetch artifacts" pipeline. You want this pipeline separate from your main pipeline because you are less likely to destroy it and it can work in parallel to the master pipeline

fly -t control-tower-pautomation set-pipeline -p fetch-artifacts -n   -c ~/ops-manager-automation-cc/ci/fetch-artifacts/pipeline.yml -l ~/private.yml 

fly -t control-tower-pautomation  unpause-pipeline -p fetch-artifacts

Run the master pipeline with the commands below:

fly -t control-tower-pautomation set-pipeline -p ${PRODUCT_SLUG} -n \
  -c ~/ops-manager-automation-cc/ci/${PRODUCT_SLUG}/pipeline.yml \
  -l ~/private.yml

fly -t control-tower-pautomation unpause-pipeline -p ${PRODUCT_SLUG}

This should begin to execute in ~60 seconds.

Be aware that you may be required to manually accept the PivNet EULAs before a product can be downloaded so watch for pipeline failures which contain the necessary URLs to follow.

You may also observe that on the first run, the export-installation job will fail because the Ops Manager is missing. Run this job manually once the install-opsman job has run successfully.

Teardown

You have a "delete-installation" pipline that can be setup with this command:

fly -t control-tower-pautomation set-pipeline -p delete-installation -n \
  -c ~/ops-manager-automation-cc/ci/${PRODUCT_SLUG}/delete-installation.yml \
  -l ~/private.yml

fly -t control-tower-pautomation unpause-pipeline -p delete-installation

You will need to manually run this pipeline.

The following steps will help you when you're ready to dispose of everything.

Use the om tool to delete the installation (be careful, you will not be asked to confirm this operation):

om delete-installation

Delete the Ops Manager VM:

gcloud compute instances delete "ops-manager-vm" --zone "${GCP_AZ1}" --quiet

Unwind the remaining PCF infrastructure:

cd ~/terraforming/terraforming-pcf
terraform destroy --auto-approve

Uninstall Concourse with control-tower:

GOOGLE_APPLICATION_CREDENTIALS=~/gcp_credentials.json \
  control-tower destroy \
    --region ${GCP_REGION} \
    --iaas gcp \
    pautomation

If you just want to destroy the current installation and "start clean" you can run this command that would teardown the entire PCF environment and reset the pipeline:

om delete-installation && \
gcloud compute instances delete ops-manager-vm --delete-disks all --zone ${GCP_AZ1} --quiet && \
cd ~/terraforming/terraforming-pcf && \
terraform destroy --auto-approve && \
terraform init && \
terraform apply --auto-approve && \
sleep 70 && \
credhub set -n pivnet-api-token -t value -v "${PIVNET_UAA_REFRESH_TOKEN}" && \
credhub set -n domain-name -t value -v "${PCF_DOMAIN_NAME}" && \
credhub set -n subdomain-name -t value -v "${PCF_SUBDOMAIN_NAME}" && \
credhub set -n gcp-project-id -t value -v "$(gcloud config get-value core/project)" && \
credhub set -n opsman-public-ip -t value -v "$(dig +short opsman.${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME})" && \
credhub set -n gcp-credentials -t value -v "$(cat ~/gcp_credentials.json)" && \
credhub set -n om-target -t value -v "${OM_TARGET}" && \
credhub set -n om-skip-ssl-validation -t value -v "${OM_SKIP_SSL_VALIDATION}" && \
credhub set -n om-username -t value -v "${OM_USERNAME}" && \
credhub set -n om-password -t value -v "${OM_PASSWORD}" && \
credhub set -n om-decryption-passphrase -t value -v "${OM_DECRYPTION_PASSPHRASE}" && \
credhub set -n domain-crt -t value -v "$(sudo cat /etc/letsencrypt/live/${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}/fullchain.pem)" && \
credhub set -n domain-key -t value -v "$(sudo cat /etc/letsencrypt/live/${PCF_SUBDOMAIN_NAME}.${PCF_DOMAIN_NAME}/privkey.pem)" && \
credhub set -n region -t value -v "${GCP_REGION}" && \
credhub set -n az1 -t value -v "${GCP_AZ1}" && \
credhub set -n az2 -t value -v "${GCP_AZ2}" && \
credhub set -n az3 -t value -v "${GCP_AZ3}" && \
credhub set -n rabbitmq-multitenant-password -t value -v "${RABBITMQ_MULTITENANT_ADMIN_PASSWORD}" && \
echo "---" > ~/state.yml && \
gsutil cp ~/state.yml gs://${PCF_SUBDOMAIN_NAME}-concourse-resources/ && \
fly -t control-tower-pautomation destroy-pipeline -n -p ${PRODUCT_SLUG} && \
fly -t control-tower-pautomation set-pipeline -p ${PRODUCT_SLUG} -n -c ~/ops-manager-automation-cc/ci/${PRODUCT_SLUG}/pipeline.yml  -l ~/private.yml

fly -t control-tower-pautomation unpause-pipeline -p ${PRODUCT_SLUG}

If you have some leftover resources, you can clean them up (interactively) with the command below.

WARNING: This is a dangerous command that can wipe out all the resources on your GCP project!

bbl cleanup-leftovers \
  --gcp-service-account-key="$(cat ~/gcp_credentials.json)" \
  --gcp-region=${GCP_REGION} \
  --iaas gcp