LRPs in k8s Spike Option 3 from previous Spike

[andrew-edgar]

See previous spike and try out option 3 from story #30

[mnitchev]

Update on the spike

TL; DR

Looks like the changes should be fairly straightforward for the purposes of the spike. [UPDATE] We have made the changes to CC and Eirini locally but need to test them since they probably (most definitely) will break something. We still need an environment where we can deploy CC and Eirini so we can test.

Challenges

LRPs in K8s:

To complete the story k8s/list.go needs to be implemented together with an lrp handler.
[UPDATE] It was easier to actually have a List method in Desirer since it already has a kube client set up.
For start endpoint Desirer should be used.
Listing kube deployments is pretty simple. The following is all we need (excluding mapping kube Deployment to LRP):

client.AppsV1beta1().Deployments(namespace).List(av1.ListOptions{})

Using process (from CC) in our api

• on the Eirini side we need::

        Name            string
        Image           string
        Command         []string
        Env             map[string]string
        TargetInstances int

For example the eirini handler could exepct the following json in its endpoint:

{
    "name": "appname_guid",
    "image": "url?",
    "command": ["ls", "-l", "-s"],
    "env": {
        "GOPATH": "/go",
        "PORT": "8080"
    },
    "targetInstances": 4
}

• in CC we have the ProcessModel object that describes LRPs. It seems like it has the following attributes, but since these come from the databse, this should be investigated further:

:name, :production, :space_guid, :stack_guid, :buildpack,
:detected_buildpack, :detected_buildpack_guid,:environment_json, :memory, :instances, :disk_quota,
:state, :version, :command, :console, :debug,:staging_task_id,
:package_state, :health_check_type, :health_check_timeout,:health_check_http_endpoint,
:staging_failed_reason, :staging_failed_description, :diego,:docker_image, :package_updated_at,
:detected_start_command, :enable_ssh, :ports

In our case :name, :docker_image, :command, :environment_json, :instances (or the method desired_instances) seem like all the things we need.

• [UPDATE] Listing LRPS needs to conform to the Diego model for the spike to work. It looks like the only thing that is needed is the process_guid since its the only field used in ProcessSync. Returning something similar to the following json looks like it would be enough not to break anything:

{
    "desired_lrp_scheduling_infos": [
        {
            "desired_lrp_key": {
                "process_guid": "the name of a kube deployment",
            }
        },
        {
            "desired_lrp_key": {
                "process_guid": "the name of another kube deployment",
            }
        }
    ]
}

Implementing the OPI Client

• AppRecipeBuilder invocations in ProcessesSync should be moved in Diego::Client or in BBS Stager/Apps/Task clients. For the spike we simply removed the calls to AppRecipeBuilder and pass directly ProcessModel to our client.

• [UPDATE] OPI::ApsClient replaces BbsAppsClient and is instantiated in the DependencyLocator in its place. This seems to be the simplest way to make the change. Our client would have to implement the following methods:

def desire_app(lrp)
def fetch_scheduling_infos
def update_app(process_guid, lrp_update) #should be a no-op
def get_app(process_guid) #this one should return ::Diego::Bbs::Models::DesiredLRP since DesireAppHandler depends on it
def stop_app(process_guid) # this too should be a no-op
def bump_freshness #looks like it isn't used too often, but its probably safer to just stub it 

def stop_index(process_guid, index) #probably safe to ignore this one for now, only used in messenger

• [UPDATE] the method fetch_scheduling_infos returns Bbs::Models::DesiredLRPSchedulingInfosResponse in its body, which contains several other nested objects. Fortunately it does seem like only one field is used — process_guid in ::Diego::Bbs::Models::DesiredLRPKey which should be the name of the K8s Deployment. So our endpoint should return a json similar to the following one:

{
    "desired_lrp_scheduling_infos": [
        {
            "desired_lrp_key": {
                "process_guid": "deployment_guid_1"
            }
        },
        {
            "desired_lrp_key": {
                "process_guid": "deployment_guid_2"
            }
        }
    ]
}

Changes that need to be discussed for the real implementation

1. We should keep both Diego and OPI as options in the DependencyLocator, so maybe we should use a FeatureFlag for determening which one to use? The bits service does something similar.
2. Replacing Diego::Client or each of the BBS Clients are the two options. In the long run maybe a middle ground would be best, since we want to properly abstract scheduling and there are Diego specific dependencies that have to be removed in both approaches. For now though doing a proper implementation of the above OPI Client should be enough to complete the next stories.
3. Since we control the CC Client we are free to do whatever we need with the Eirini LRP API, so we should decide what it will look like.