Aug 9, 2026

Open-Source Oil & Gas?!

platform-engineeringopen-source

2026-08-09 14:21 +0000

So, I’m back. Life got so hard I haven’t written a single blog post in more than a year. Well, to be honest, I did try multiple times, but never finished a single one. University got pretty hard, and then I went and accidentally got a job. Now, in this job market, you might be angry as to how one gets a job accidentally. But, truly, it just happened.

And with this job is how we get to the subject of this blog post: OSDU.

What Even Is OSDU?

Well, OSDU (Open Subsurface Data Universe), is, as the name implies, a data universe. Okay, it’s not a universe, but it’s a platform. A platform that is made up of a ton of services. Its job? To keep and ingest oil and gas data (and some other data too). Right now there’s a new version rolling out – M27 (or 0.30) – which I’m firefighting for a certain CSP (Cloud Service Provider).

Counting how many services OSDU is comprised of depends on what you want to use. As of writing, there are 29 graduated services, and a total of ~32 services. However, only 9 of those are, what are called, “core” services. These core services are the foundations that the entire platform is built on, and they are:

Now, as I work on pipelines/platform engineer stuff, I don’t know in detail what exactly each of them does. But, what I can do, is assume from my experience:

  • Partition – Basically the baseline for everything. It creates partitions or retrieves them. Partitions are where your data lives, so this service is pretty important.
  • Legal – You’re working with oil and gas data, that has legal complications. That’s exactly where the Legal service comes in to save the day. It creates and manages legal tags that you assign to data.
  • Entitlements – The RBAC of OSDU. It manages permissions with groups.
  • Schema – Validator of the data you put into OSDU. You can add your own schemas or use the provided defaults.
  • Storage – You might think this stores files, and you’d be wrong. Obviously, you wouldn’t store files on a service, but what it does is keep metadata. And to be more exact, it manages the entire life-cycle of that metadata.
  • Indexer – Indexes data into Elasticsearch. Pretty simple.
  • Search – Searches through the indexed data. Helped by Indexer.
  • File – Manages requests for file uploads or locations.
  • Ingestion Workflow – A wrapper around Airflow’s API with some OSDU related middleware on top.

And these 9 services are just the tip of the iceberg of what OSDU can actually do. So much so, that even I don’t know.

Junior’s Experience With OSDU

It’s complicated as hell. As someone who lived, and still lives, in a “I have no idea what’s going on right now” type of environment, I picked up the basics quite quickly. But, honestly, this project would be so overly complicated for any generic junior just starting out their career. Which is a reason as to why I’m writing this. For anyone, be it a junior or a senior Platform/Systems/DevOps Engineer getting into this project, it will be a struggle at the start. Especially, figuring out the open-source aspect of the project. What I learned about OSDU was mostly through my own trial and error, which a lot of people might not be accustomed to.

Of course, you can read through official documentation, READMEs, or whatever else. But, in my opinion, for someone who will be working on upkeeping those repositories, it won’t be enough. And neither will this, you, quite literally, have to get first-hand experience to truly understand it.

Now, I don’t know the Software Engineer’s side of OSDU. I can only write this from a Platform Engineer’s point-of-view, and even that only from Google Cloud’s side. So, take whatever is written here with a grain of salt.

Pipelines Are Cool

The CI/CD Pipelines repository is a huge collection of shared GitLab CI pipeline files/jobs, where I might have contributed a not so overly complicated job validating OpenAPI specs. That single job runs across ~25 services, is a mandatory gate, and has taken most of my sanity away. But, I digress.

The pipelines are split into two types: standard and trusted pipelines. The standard pipelines are barebones testing if your code compiles, and passes tests. For some repositories, it’s just linting and nothing more. Where stuff gets interesting is with trusted pipelines. Now, to run trusted pipelines, you need Maintainer permissions. If you have them – perfect, if you don’t – ask someone who has. Afterwards, create an MR, and another branch with the same name, but with a trusted- prefix, and watch heaven’s gate open up. In this case, “heaven’s gate” is a pipeline which runs for ~3 hours (I wish I was lying). Of course, how long it takes depends on the service and the amount of jobs the pipeline spawns. But, you should know it’s almost never quick.

What is cool about these pipelines is that it runs everything. Well, almost everything. Of course, release pipelines are separate, especially for Google Cloud. But, what you will see is your changes getting deployed in every CSP’s and Community’s environments. And to access the Community environment, you can use https://osdu.dev1.osdu-cimpl.opengroup.org/ as your API host. For example, https://osdu.dev1.osdu-cimpl.opengroup.org/api/schema-service/v1/swagger.

As mentioned above, there are also release pipelines. Two to be exact: release/* branch pipelines, and tag pipelines. These pipelines mostly look identical, but you will notice some additions from Google Cloud’s side, like GCR publishing and deployments to their preshipping environment. There’s nothing too wrong with creating your own release pipelines for testing if you have the permissions, but you should try not to mess up other CSPs or the Community in the process.

Every service/repository tries to respect these shared pipelines as much as they can, but these pipelines aren’t made to fit everything. Most, if not all, repos that you’ll find will have overrides in their respective devops/<CSP>/ directory (for example, here’s an override file for Google Cloud on Schema). If you ever work on overrides, always remember: if there’s an error that can be generic, it is generic and happening in other overrides.

However, there exists a service that does something that’s so special that it requires its own section…

Seismically Confusing Pipelines

Seismic – or more formally the Seismic Store Service, is an interesting case when talking about pipelines. At an initial glance of its repo, it doesn’t look like anything special. Moreover, you might think it isn’t even that interesting. But, the deeper you dig, the more fascinated one gets.

Starting off, unlike many other services, which are written in either Java or Python, Seismic is written in TypeScript. It is a service which is classified as a Domain Data Management Service – or DDMS for short. As seismic data is quite important, you can also assume that this service is also important, even if it’s not a core service.

The first step Seismic takes for pipelines is to divide them up even more. Now, instead of only having standard and trusted pipelines, you also have a no-detached-pipeline. To be more exact, no-detached-pipeline is a label that you can place on your MR to run a much simplified version of a trusted pipeline. One cool part about it is that you don’t need Maintainer permissions to run them. I’ll let you check out the differences instead of explaining them: normal pipeline, trusted pipeline, no-detached-pipeline.

When looking at those pipelines, you might have noticed something off. The names of the jobs are prefixed with filemetadata, sdms-v4, or sdms. Why? Well, welcome to your first realization: this repository holds more than a single service. To be exact it holds version 3 and 4 of Seismic, and filemetadata. For filemetadata, I should say used to hold. As of M27/0.30, it’s deprecated and fully removed. You might be asking why there are two versions of the same service in the same repository, and, to be honest, I don’t have an answer to that.

Now, that you understand that, I’ll invite you to take a look at the pipeline definitions. In almost every file, you’ll see the same pattern:

# Anytime you see CImpl, it means Community Implementation
cimpl-deploy:
  needs: []
  extends: []
  rules:
    - when: never

You might also recognize the names of the files, as they are direct copies of the files in the shared pipelines. Everything in this directory disables the jobs defined in the shared pipelines, and then redefines them in each service. You can take a look at Seismic v3’s overrides here. Here’s the redefined Community deploy job:

sdms_cimpl-deploy:
  extends:
    - cimpl-deploy
    - .cimpl-variables
  needs:
    - 'sdms_cimpl-containerize-gitlab'
    - 'sdms_cimpl-helm-charts-gitlab'
  variables:
    CIMPL_SERVICE: seismic-store
    CIMPL_HELM_DEPLOYMENT_DIR: "app/sdms/devops/core-plus/deploy"
    CIMPL_HELM_TIMEOUT: "--timeout=1200s"
  rules:
    - if: $CI_COMMIT_BRANCH =~ '/^release/'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CIMPL == '1'
      when: on_success
    - if: $CI_COMMIT_BRANCH =~ '/^trusted-' && $CIMPL == '1'
      when: on_success
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CIMPL == '1'
      changes:
        - devops/**/*
        - app/sdms/**/*
      when: on_success

If you compare this with the original cimpl-deploy job, you’ll see there aren’t a lot of changes, mostly to do with variables and rules. But, this pattern is forced because of how the repository is structured to have multiple services in a single repo. So, if you ever bring a new job in the shared pipelines, that you also believe should be in Seismic, you have to go through this.

Problematic Pipelines

Having a shared pipeline repository is great, when you can actually manage it as a platform, instead of a bunch of overrides patching up what’s fundamentally a problematic structure. I’ll draw two anecdotes from my own experience: GC’s dev2 environment removal, and Google’s removal of docker from their cloud-sdk images.

Removing of the dev2 environment overrides was one of my eye-opening realizations of the tangled mess these overrides are. For some repos, you’ll see needed overrides, for some just rules disabling them (with/without empty scripts). This simple environment removal took 50 MRs over 50 different repositories. However, this isn’t to say that the shared pipelines are horrible, but to show the scale that they can exist in. Just because there are ~32 services, doesn’t mean there aren’t Python libraries, Airflow DAGs, and more to still manage.

Google’s removal of docker from cloud-sdk images has been one of the biggest problems in the pipelines as of late. All because we use cloud-sdk:alpine, instead of a pinned version. Now should we use a pinned version everywhere? No, because each update then would take those same 50 MRs per upgrade. At the end, it will be such a waste of time, that it won’t be allocated for anymore. But, when thinking about it, I came up with an idea of what a fix for it could look like:

.gc-cloud-sdk-image:
    image: gcr.io/google.com/cloudsdktool/cloud-sdk:579.0.0-alpine

A simple template that adds the cloud-sdk image to any job that extends from it. It can be provided in the shared pipelines acting as the sole source-of-truth of its version in any job. So, if you ever needed to update it, you have a single place, instead of 50. But, this doesn’t fix the problem with the missing docker binary. Should we just never update now? Well, for this we can be a little more pragmatic when designing the pipelines, and create a preliminary script, like:

.gc-cloud-sdk-job-prelim:
    before_script: []

gc-any-job:
    extends:
        - .gc-cloud-sdk-image
        # Don't extend `.gc-cloud-sdk-job-prelim` as we might have to add more
        # stuff to `before_script` and may end up accidentally overriding it.
        # We are just making it super obvious if someone copies this over.
    before_script:
        # !reference is a GitLab CI specific concept, it basically copies from
        # a job a specific field, in this case the above before_script.
        - !reference [.gc-cloud-sdk-job-prelim, before_script]

As such, anything that overrides the original job will be able to copy that preliminary script to ensure no updates to the image itself break it. But, of course without it happening to you, you’ll almost never get to thinking about this. You kind of see now why having experience as a Platform/DevOps engineer matters.

Also, if whenever you are designing a template and the name you chose for it has the word “and” in it, you are doing something wrong unless it’s a combination of two other templates.

I Wanna Deploy!

If you looked around the repositories, you might have seen directories like devops/core-plus/deploy. These directories hold Helm charts that describe how to deploy each service (for example, here’s Schema’s Helm chart).

However, for many, be it a beginner in OSDU or an advanced user, deploying every service like this is tedious as hell. This is why most of the time, you would either try to test your changes on the community by calling the API, or deploy your own environment using a CSP implementation, for Google Cloud this would be the Deployment Engine. But, recently, there has been a development in this direction. More specifically, a CLI has been vibe-coded/agentically engineered, called cimpl-stack.

CImpl Stack is a Python CLI developed to wrap around a bunch of Flux manifests/kustomizations to deploy a development OSDU environment in every possible way. You can deploy to a local KinD cluster, Azure, AWS, GC, OpenShift, or whatever K8s cluster you are connected to. However, the coolest thing about it is how simply you can do it:

# Check if you have the necessary dependencies to deploy
cimpl check --provider=gcp

# And deploy with a single command
cimpl up --provider=gcp

# Check the deployment status
cimpl status

# Get endpoints and credentials
cimpl info --show-secrets

# Finally, kill the deployment
cimpl down --provider=gcp

You can also specify what you want to be deployed with the --profile flag, which accepts:

  • minimal – No OSDU services, just dependencies like Elastic, Redis, RabbitMQ, etc.
  • core – Deploys the 9 core services listed at the beginning of the blog.
  • core-plus – Deploys 15 more services on top of core, like Dataset, Notification, Secret, GCZ, Unit, etc. Not including DDMS.
  • graduated – Deploys every DDMS on top of core-plus.
  • full – Deploys sandbox services, like Config and DSPDM.

Overall, if you want to try out OSDU, CImpl Stack is the simplest route. However, if you like to get your hands dirty with Terraform, you can use Google Cloud’s Deployment Framework, which is divided into two modules: infrastructure module, and K8s module. To use it, you can follow the official wiki page for it, as you’ll need to create files and configuration based on current versions. However, do expect that to be outdated as well, as such, experiment and have fun!

Composite Ending

As you can see, OSDU is a world of both interesting and confusing ideas that neatly exist in a single GitLab instance. Everything you have read is coming out of the mere 6 months that I have been a part of it. It has truly been an experience like no other could be. To me it is comparable to how I felt when learning and actually understanding Haskell, which was truly the moment I felt like I understood what designing software really was.

When learning at the start, it never really clicks into place, you think you understand it, but given a different environment, much like ML models, you feel lost. Even though at that time I had worked on low-level stuff with C, Haskell was truly that point, where everything started making sense. Maybe my setup of Linux From Scratch (LFS) also had something to do with it.

Working on OSDU as my first ever job was that experience. DevOps concepts, like K8s, Helm, pipelines, and GitOps don’t really click into place when you are repeatedly writing the same glorified NGINX templates. They do when you use them in practice, in a complex system. These last 6 months were exactly that. Not only with OSDU, but with my Bachelor’s project for which I experimented with IaC, writing GitHub Action workflows for full CI/CD, deploying a K8s cluster using Talos, setting up ArgoCD for GitOps, and exposing the cluster using Cloudflare Tunnel (you can check that mess out here).

Overall, if for whatever reason you are in the same position as I am/was, just know: it is hard, and you’ll get used to it.

You are now entering a world from which you will never return
You may choose to embrace it
Or fight it, but you cannot change it
Nor can you go back

How you choose to feel will determine how you feel
Accept this willingly, and you will be enlightened
Fight it, and you’ll be damned
But, it is your choice to make

You Are the Miserable — t+pazolite