Highlights of OpenStack PTG, Denver 

OpenStack foundation organizes “Project Teams Gathering” every six months, mostly in Q1 and Q3, at the beginning of the development phase of a release cycle. Previously known as “Design summit”, the main motive for this event is to discuss the upcoming cycle, solutions for complex issues, and innovative solutions to critical items. I was privileged to attend the Denver PTG this September.

It was my first time joining the Project Teams Gathering (PTG) since it started in Atlanta. The location of the event was pretty unique in its own way. I had never been to this part of USA and I could feel the difference. The event was held in the Renaissance Denver for 5 days between September 11 and 15. I arrived here on Sep. 10 and I could already see the active contributors in the hotel lobby discussing stuff related to the latest technology trends in the space. I met some of the friends and took rest for the rest of the day.

Day-1, September 11

The day started with registration for the event. I joined the openstack-ptg channel to get the updates about the day and there I was introduced to ptgbot. Initially many people including me were a bit confused with how it works, but as we got familiar with it, we got more used to it for tracking the events. As per schedule, the first two days of the event were dedicated to inter-project discussions.

I headed directly to the infra/stable/release/requirements room for discussions of requirements team. We had a discussion around topics to be worked on in Queens which included the per-project/independent/divergent requirements, OpenStack client testing, and Python 3. The discussion was pretty good with insights provided by Tony Breeds, Matthew Thode, Dirk Mueller, Monty Taylor, and John Dickinson. Post lunch, I joined Kolla team with discussions around collaboration across different deployment tooling in OpenStack. We had discussions around architecture, health monitoring, the role of containers, Kubernetes, and security. I also attended the Technical Committee (TC) meeting for Rebooting of the Stewardship WG and Onboarding new community members.

The day ended with unofficial PTG happy hour at the elevated lounge in Renaissance Denver.

Day-2, September 12

The day started with meeting people at registration desk hallway. I joined the TC discussion for Q & A around new projects in OpenStack. The discussion included emerging projects like Blazar, Glare, Gluon, Masakari, Stackube, Cyborg, and Mogan. Each project representative presented TC members with a general project overview, objectives, current status, and tried to answer any queries related to the Requirements for new OpenStack Projects applications. TC members also asked details related to the maturity of projects with respect to contributor’s diversity, the potential to increase collaboration, identifying any overlap with current/existing projects in OpenStack. We also had a discussion around potential project removals from OpenStack official projects. This is due to very low contributor activity during the cycle, no project goals achieved or changed focus of the participating organizations. The particular projects highlighted in the area are Searchlight, Solum, Designate, CloudKitty.

Michal Jastrzębski and I were chasing the infra/TC members for resolution related to publishing kolla images on the public registry like Quay with openstack-kolla namespace. We did not get any direct solution and I think there is a need to initiate a discussion on openstack-dev mailing list for further discussion.

I met with Saad Zaher (Project Technical Lead) of Freezer regarding the current status of the project. I had a very informative discussion and got a clear idea of things to do.

We had few hallway discussions related to kolla-kubernetes, openstack-helm, kolla with Ironic, kolla with neutron.

In the meantime we also had team photos clicked for Kolla and requirements team. This was the end of the first section of schedule of inter-project discussions. The day ended with IBM sponsored happy hour in the hotel where random discussions happened regarding projects, OpenStack, etc.

Day-3, September 13

Day 3 of Queens PTG started with project specific design discussions, I joined the Kolla team where we started off with the design summits we had and a very important thing for the community, the "Documentation". We broke down the discussion in the documentation for quick-start with kolla contributor, operators, and reference documentation. The documentation available currently is scattered across projects after project split and it’s essential to have a common landing page on OpenStack Deployment guides which everyone can refer to. We had representatives from the Documentation team Alexandra Settle, Doug Hellmann, and Petr Kovář who are working on improving the doc experience by migrating the docs to a common format across the community. They understood the problem Kolla is facing and we had a discussion where we created the table of contents for all available and required documentation for Kolla.

Kolla team then joined the TripleO team which is consuming the kolla images for OpenStack deployment for a discussion about collaboration of efforts. The teams will work together to improve the build and publish pipeline for kolla images, improving & adding more CI jobs for the kolla/kolla-ansible/kolla-kubernetes, configuration management post deployment of containers. The tripleo team has come up with basic health-checks for containerized deployment, the Kolla team will help get those checks in the current kolla images and improve on those to better monitor the containerized OpenStack deployment. The teams will also collaborate on improving the orchestration steps, container testing, upgrades, and creating metrics for OpenStack deployment.

During lunch, we had extended discussion with Lars Kellogg-Stedman and Kevin Fox around “Monitoring OpenStack with Prometheus and other monitoring tools”.

Post lunch, Kolla team started with key discussion to the heart of operators, the OpenStack plugins deployment with Kolla. There are multiple issues currently related to the plugin like when would be an ideal time to make them available, during build/deployment, non-matching dependencies to OpenStack components for plugins, etc. The team came up with multiple permutations of options available which would need to be PoCed during the release.

Since the inception of project loci, there has been discussion around kolla-images size and the team had an interesting discussion on how to reduce that. The important part is to remove the things like apt/yum cache, removing the fat base image, etc. The team also discussed utilizing alternate container build tooling to write own image build tool. The team will hack on Friday, September 15, removing the fat base images and see if that improves the image size.

External tools like Ceph are common pain points when we are doing OpenStack deployment. When Kolla community evaluated the options for Ceph as storage backed for containerized OpenStack deployment there wasn’t a thing like containerized ceph. The team built it from scratch and got it working. The ceph team has currently come up with ceph-docker and ceph-ansible. It would be useful for operators that Kolla uses the tools directly available from vendors. We had a discussion with representatives from ceph to initiate the collaboration to deprecate current ceph deployment in Kolla and use the combination of ceph-docker & ceph-ansible. It will help both the communities and will benefit exchange of things at each end.

I got a surprise gift of vintage OpenStack swag from the PTG team

The day ended with hanging out with Kolla team-mates at “Famous Dave's”.

Day-4, September 14

Day 4 of PTG started with next Kolla discussions related to kolla-ansible. Discussion started with kolla dev-mode effort by Paul Bourke. The discussion was about current missing pieces in dev_mode like installing clients, libs, and virtualenv bindmount. The goal of the cycle was to fill the missing pieces, verify options for multi-node dev_mode, investigate on options for remote debugging and also consider using PyCharm.

One of the important topics in Kolla is the gating. Currently, Kolla has around 14 different gates for deployment testing and it has to be improved with testing the deployment for sanity with Tempest. This will help to validate the entire deployment in the gates. Upgrades testing is also one key requirement, Kolla team will model something like grenade testing for it. The key is to maximize the testing of scenarios that Kolla supports in gate, but since we are restricted with openstack infra resources as well as the time each test takes to validate. It was agreed that team members will create a list of scenarios and assign them to everyone to verify and record the results in a central location. This will also help to evaluate the stability of Kolla deployment in each release.

Skip level upgrades was one of the major talking points in the current PTG. Kolla team will evaluate fast forward upgrades for each service deployed with Kolla to decide on skip level upgrade support in Kolla. This would be a PoC in current cycle.

The second half of the discussion was around the kolla-kubernetes, where the team discussed the roadmap for the current cycle. That will include upgrade prototyping for z stream & x stream services, validate the logging solution with fluent-bit, automated deployment, remove deprecated components and improve documentation.

Most of the teams had wrapped up their design discussions on Thursday and were gearing up for “the hackathons” on the last day.

Day-5, September 15

Day 5 of PTG started as the day for hackathons, and general project/cross-project discussions. Most project teams were preparing for their travel plans or site-seeing in Colorado. The Kolla team started the day with alternate Dockerfile build tool review. Later on the day, there was something everyone in OpenStack and containers community was looking forward to - the OpenStack - Kubernets SIG with Chris Hodge. Some key targets for the release were identified including contributors interested. We then had the most pressing issue for all deployment projects based on containers - the build and publishing pipeline for kolla images with openstack-infra team. Most of the current requirements, needs, and blocking points were identified for rolling this feature. The Kolla team and OpenStack infra team will work together to get this rolling in the starting phase of this cycle once zuul v3 rollout becomes stable. The Kolla team ended the day early for some much needed buzz for the whole week's work at “Station 26”.

It was a very insightful session this time around in Denver. And with the kind of work Opcito is coming up within OpenStack space, I am already looking forward to the Dublin edition of OpenStack PTG.