Speaker: Lynn Owens
For more, see the table of contents.
- Every gitlab project has own .gitlab-ci.yml file. Great for getting started
- Quickly have hundreds of projects
- Goal is to eliminate copy/paste by centralizing in a few projects
What NAIC has
- 200+ projects maintained by 11 teams in 2 dev orgs
- Pipeline is inner source
- Version 6 of pipeline; working on version 7
- Reduced maintenance burden by making change once and not in each project
- Hosted directly on gitlab.com
Milestone 1 – Hidden jobs for pipeline project
- GitLab has “hidden” jobs
- Start with a period
- Don’t appear in any pipeline; just for the common code
- The “pipeline” project has a .gitlab-ci-base.yml which contains common code
- Common code makes no assumptions about teams and is configurable for all known use cases
- v1 was about two dozen lines of common code
- The client projects include the pipeline code (can include in any part of gitlab so doesn’t need to be yours)
include: -project: 'NAIC/pipeline' -file './gitlab-ci-base.yml'
- Then added jobs that extended the hidden jobs to call functions in the base code. Where deploy_foo is in the base code
deploy_foo: stage: deploy extend: .deploy_s3 variables: ...
- Advises against pinning the pipeline to a tag because don’t get bug fixes and everyone has to upgrade manually
- Don’t include stages in the pipeline as it forces one opinion on everyone. Many groups had written a pipeline for their use case and not all same.
Milestone 2 – Profiles
- Found a half dozen use cases. ex: Maven for Java, NPM building Angular etc.
- The .gitlab-ci.yaml was a copy/paste of the others in the use case.
- Made profiles/maven-java.yml and the like in the common profile
- Profiles are not one size fits all because there are a bunch of different ones and can still use the milestone v1 approach.
Milestone 3 – Pipeline scripts
- Common code like logging, calling rest apis, etc
- Switched from bash scripting to python so had common code in modules and could unit test the modules
Options to get scripts
- Could have the pipeline create a tar.zip and upload to a repo. This is a little slow
- Could have a global before_script that does a git clone of peipleine-scripts. Uses a network connection
- Could bake the scripts into an image. Requires a pipeline
If was doing again, wouldn’t create separate pipeline-scripts because tightly coupled to pipeline. Doesn’t change problem of using the scripts though.
- If client projects are all using the default branch, small changes will affect them all.
- Use a testing framework for script code (ex: python/go)
- Follow development practices
- Write a sample app for each profile. Have the common pipeline trigger a downstream pipeline on this project. For any merge to master, the downstream jobs must pass.
- Before major refactors, inventory profile jobs and audit afterwards,
Milestone 4 – Profile Fragments
- Had about 24 profiles (ex: maven-java-jar, maven-java-pom, maven-java-k8s, etc)
- Typically three components – build tool, language, deployment method
- These profiles had a lot of copy/paste
- Decomposed into fragments – ex: maven, npm, java, angular, k8s, s3)
Selling the idea
- Needed to convince people to use this pipeline instead of writing own or another team.
- Offer flexibility
- Show value
- Follow semantic versioning to the T (he tags every merge to master of the pipeline even though encourages use of the default branch. the tags are good rollback points or if the project needs something older)
- Changelog everything
- Document well
- Train and evangelize
- Record training so have library
This was a good case study and useful to see concrete examples and techniques. I wish we could see the code, but I understand that belongs to their org.