I’m building GreyhoundDash, a monitoring dashboard that can run either self-hosted or as SaaS.

The dashboard is intended as a tool to help users monitor their self-hosted applications from a single view. This is definitely not a unique endeavour as there are many others like Heimdal. But it serves as an interesting platform to base a few different experiments from.

To recognise and health-check the long tail of open-source, self-hostable services, the dashboard needs a machine-readable definition for each one.

Hand-authoring those definitions doesn’t scale; the Awesome-Self-Hosted index alone lists hundreds of candidates.

Instead of grinding through them manually, I’m using large-language-model code generation to create the boilerplate classes automatically. This series documents the experiment—why I chose the approach, how the tooling is wired together, and what still needs tightening up.

Requirements

Core workflow

1. Detect new service

   Scan the curated list (JSON produced from Awesome-Self-Hosted).

2. Generate code

   Produce a Python subclass of ServiceBase that performs a simple “is-alive” check.

3. Create PR

   Commit the file on a new branch, push, and open a pull-request back to main. Branch/PR names must be traceable (feat/basic-svc-<slug>, etc.).

Determinism & safety

• Same input → same output. If the model drifts, retry up to n times, then quarantine the service.

• Validate the JSON schema with Pydantic before writing to disk; on failure, log and skip.

Cost & noise controls

• Batching: group 10 services per PR (PR_FREQUENCY) so reviewers aren’t flooded.

• Budget guard: stop once the estimated OpenAI spend or token quota for the run is hit.

• Checkpointing: persist a completed_services.json so reruns pick up where they left off.

These constraints keep the process affordable and the repository maintainable while still letting the LLM do the grunt work. This article will focus on the first and third points.

Harvesting candidate services

Source: https://awesome-selfhosted.net/#software
Goal: Iterate over every entry and convert to a JSON format for easy parsing

This bit was relatively straight forwards, I decided to parse the content on this page https://awesome-selfhosted.net/#software (awesome project that I fully endorse) using the BS4 lib. I won’t go into specifics here but I will share the code:

Extraction pipeline:

1: Fetch the HTML

def fetch_awesome_selfhosted():
    url = "https://awesome-selfhosted.net/#software"
    response = requests.get(url)
    if response.status_code != 200:
        raise Exception(f"Failed to fetch the page: {response.status_code}")
    return response.text

2: Parse with Beautiful Soup

BS is a great Python library used for parsing data from web pages, often used for web scraping. Parsing over HTML can be quite delicate, as any changes to the structure of the page can throw off the code.

def extract_services(html: str) -> dict[str, dict]:
    soup = BeautifulSoup(html, "html.parser")
    services, skipped = {}, []

    for s in soup.select("#software section"):
        name = s.h3.get_text(strip=True)
        link = s.select_one("a.external-link")
        if not link:
            skipped.append(name); continue
        services[to_camel(name)] = {
            "name": name,
            "description": (s.p or "").get_text(strip=True),
            "homepage": link["href"],
        }

    if skipped:
        logging.warning("Skipped %d services without a homepage: %s",
                        len(skipped), ", ".join(skipped))
    return services

3: Persist

The list should be persisted so it can be used as input to the Language Model.

with open("tooling/awesome_selfhosted_services.json", "w") as f:
    import json
    json.dump(result, f, indent=2)

The scraper gives me a one-shot JSON cache with exactly the three fields the generator needs—name, description, homepage. Whenever Awesome-Self-Hosted changes, I just rerun the script and the new entries appear in the file.

It’s deliberately minimal:

• No retry/back-off and no DOM diffing—acceptable for tooling, not for prod.

• Relies on the current page structure; if the maintainers rename a class, the selector fails and I’ll fix it then.

• Runs on demand, outside the dashboard’s runtime path, so a crash can’t hurt users.

I also persist a awesome_selfhosted_services_completed.json checkpoint. Each code-gen run looks at that file first, skips anything already merged, and works on the remainder. Preventing duplicate processing and allows the code to pick what hasn’t been done only.

Generating the service stubs

The LLM must return something I can deserialize without guess-work, so the interaction is schema-first:

1. Schema contract
   A single ServiceResponseSchema (Pydantic) drives both the OpenAI call and local validation:

    class ServiceResponseSchema(BaseModel):
        file_name: str      # e.g. "service_grafana.py"
        service: str        # Human-readable name
        code: str           # The Python subclass itself
        logo: str           # URL or data-URI (future feature)
        web_app: bool       # True ⇢ include, False ⇢ skip
   

   Any field mismatch raises immediately; no fragile json.loads() spelunking.

2. Prompt template
   Stored in tooling/service_generation_prompt.txt with Jinja-like tokens:

    You are ChatGPT. Generate a subclass of {{ base_class_name }}.
   
    ``` python
    {{ base_class_file }}
    ```
   
    Sample implementations of this base class:
   
    ``` python
    {{ implementation_samples }}
    ```
    Response must comply with the below schema:
   
    ``` json
    {{ json_schema }}
    ```
   

   At runtime the script simply .replace()s the placeholders—no template engine needed.

Why JSON over code blocks?
Without a rigid envelope, the model may prepend helper text or miss imports, which breaks CI. JSON gives me a single thing to parse—and Pydantic enforces it at runtime.

Why a bare string replace instead of Jinja?
The prompt is configuration, not application code; simplicity beats another dependency.

Future cleanup: swap the naïve .replace() for str.format_map() so accidental {{ }} inside comments don’t expand.

Wiring the pipeline

The automation lives in /tooling and is split into two well-defined actors:

ServiceGenerator

GithubPRCreator (excerpt)

def create_branch_and_raise_pr(self, branch, services):
    # 1. git checkout -b <branch>
    # 2. git add -A && git commit -m "feat: ..."
    # 3. git push -u origin <branch>
    pr = self.repo.create_pull(
        title=f"gen: {services[0]} → {services[-1]}",
        body="\n".join(f"{n}: {ok}" for n, ok in services),
        head=branch,
        base="main",
    )

Guard rails

• Refuses to reuse an existing branch (fast-fail).

• Logs the PR URL so CI can post a comment or Slack alert later.

ServiceGenerator (key points)

PR_FREQUENCY = 10   # services per PR
BATCH_SIZE   = 10   # services per run (keeps token spend predictable)
GPT_MODEL    = "gpt-4o-mini"

1. Prompt fill → deterministic JSON response (temperature = 0).

2. Schema validation with Pydantic; on failure retry ≤ 5 then skip.

3. Write file only when web_app is True.

4. Book-keeping — append to completed.json so reruns resume cleanly.

5. Every PR_FREQUENCY successes → delegate to GithubPRCreator.

Entrypoint

if __name__ == "__main__":
    gh = GithubPRCreator(Github(os.getenv("GH_TOKEN")), "JTSG1/GreyhoundDash")
    ServiceGenerator(openai_client=OpenAI(), github_pr_creator=gh).iterate_services()

The main block wires the two actors together; nothing else runs in global scope, which makes the module import-safe for future unit tests.

Why this structure?
A tight, two-class design keeps the surface area small: one side talks to the LLM, the other to Git. Each part can be mocked in isolation, batteries included for retries, cost limits, and PR hygiene.Wrapping up

Wrap-up

• What’s solved?
  GreyhoundDash can now turn a raw list of open-source services into ready-to-review Python stubs, batch-commit them, and open pull-requests—all without a human typing the same boilerplate thirty times.

• What’s still brittle?

  • Scraper depends on Awesome-Self-Hosted’s HTML staying stable.

  • Generated code is unit-tested only for syntax, not runtime behaviour.

  • Logo retrieval and licence compliance are TODOs.

  • OpenAI cost control is coarse (batch ceiling rather than per-token budgeting).

• Immediate next steps

  1. Add a CI job that lints each generated file and executes a dummy health-check.

  2. Swap the HTML scrape for the project’s YAML source to decouple from DOM churn.

  3. Enforce a spend cap via the OpenAI usage API and abort runs that hit it.

  4. Store prompt + schema version in git so regressions are traceable.

• Calling for help

  • Spot a service the scraper missed? Open an issue.

  • See a way to harden the GitHub automation (e.g., branch naming, PAT scope)? Let me know.

The repo is at github.com/JTSG1/GreyhoundDash. Star it and keep up to date on the development!

Over the past year, my role has increasingly involved containerization technology, particularly Kubernetes. Kubernetes is a fantastic tool for deploying applications and enabling them to scale according to demand. Once you're comfortable with these technologies, you can really start to leverage the power of elastic computing.

A supporting repo can be found here: https://github.com/JTSG1/helm-kubernetes-tutorial

Kubernetes

Kubernetes is a container orchestration platform. It allows you to take your Docker images and build production-ready workloads using YAML files. These files can be version-controlled and deployed through automated CI/CD pipelines. Sounds great, right?

Throughout this article, I'll provide some practical examples. Most of these will center around a service dashboard software called Homer. I've chosen Homer because it's relatively straightforward to get up and running, and I'm a big fan of the software for self-hosting.

Helm

Helm is essentially a templating system built on top of Kubernetes objects. It allows you to parameterize and manage releases centrally, offering greater control over your Kubernetes deployments. A great use case for Helm is the ability to tailor your application configuration to specific environments.

Imagine you've just developed the next big thing that's going to go viral and make you a fortune! If you're following good practices, you'll have several different environments set up to progress your deployment through various testing stages. Helm helps with this by allowing you to have a base configuration that you can then override for specific environments.

A Simple Kubernetes Deployment

We're going to look at a simplified set of Kubernetes manifests. Typically, in a production environment, you'd have more, but let's keep things simple for now. To deploy Homer, I want to define the following Kubernetes objects:

• deployment: This contains the actual service.

• namespace: This is used to segregate objects, as Kubernetes clusters rarely host just one workload.

• service: This defines how we expose the application to the outside world.

• service account: This specifies the 'user' the service will run under, which is especially useful when deploying to services like GCP that offer workload federation.

The file structure I'm aiming for will look like this:

./homer-sample-no-helm
    - default-deployment.yaml
    - default-namespace.yaml
    - default-service.yaml
    - default-serviceaccount.yaml

Here's the content of each of these files:

default-deployment.yaml

This object contains the actual container and its related settings.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: homer-deployment
  namespace: ns-homer
  labels:
    app: homer-deployment
spec:
  strategy:
    type: Recreate
  replicas: 1
  selector:
    matchLabels:
      app: kafka-sample
  template:
    metadata:
      annotations:
        sampleAnnotation: "a-sample-annotation"
      labels:
        app: kafka-sample
    spec:
      serviceAccountName: homer-service-account
      containers:
      # hello world container
      - name: hello-world
        image: docker.io/b4bz/homer
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi
        env:
        - name: MY_ENV_VAR
          value: "my-env-var-value"
        - name: MY_ENV_VAR_2
          value: "my-env-var-value-2"
      restartPolicy: Always

default-namespace.yaml

This is the Kubernetes namespace definition for this deployment.

YAML

apiVersion: v1
kind: Namespace
metadata:
  name: ns-homer
  labels:
    app: ns-homer

default-service.yaml

This service exposes the deployment externally using a LoadBalancer.

YAML

apiVersion: v1
kind: Service
metadata:
  name: homer-deployment
  namespace: ns-homer
  labels:
    app: homer-deployment
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: kafka-sample
  type: LoadBalancer

default-serviceaccount.yaml

This is the Kubernetes service account that the deployments will run under.

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-homer-sample
  namespace: ns-homer
  labels:
    app: home-sample

As you can already see, even for a relatively simple deployment, there are quite a few files. Maintaining these files separately for each of your RTL (Route to Live) environments would quickly become cumbersome. This is where Helm comes in handy.

Another thing you might notice is the amount of repetition across these files. A simple typo in one of these fields can lead to hours of debugging! Trust me, I've been there.

Helm to the Rescue

Now, let's look at the same set of files but within a Helm setup.

./homer-sample
    - templates/
        - default-deployment.yaml
        - default-namespace.yaml
        - default-service.yaml
        - default-serviceaccount.yaml
    - Chart.yaml
    - values.yaml

Inside the templates folder, we have the same set of files, but their content is different. We'll review that in a moment. Additionally, we also have two new files: Chart.yaml and values.yaml.

Chart.yaml contains metadata about the Helm deployment.

values.yaml is a central store for variables that we can reference in our Helm-templated Kubernetes manifests.

Now, let's re-examine the four manifest templates:

default-deployment.yaml

This file now contains references to Values.* variables, which are defined in the values.yaml file. You can immediately see the benefits of this approach to structuring these files. We are essentially templating the Helm files and injecting values defined elsewhere into them.

Also of note are the conditional statements included here.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.deployment.name }}
  namespace: {{ $.Values.namespace.name }}
  labels:
    app: {{ .Values.deployment.name }}
spec:
  strategy:
    type: Recreate
  replicas: 1
  selector:
    matchLabels:
      app: kafka-sample
  template:
    metadata:
      annotations:
        sampleAnnotation: "a-sample-annotation"
        {{- if eq .Values.conditionalAnnotations.enabled true}}
        conditionalAnnotation: "a-conditional-annotation"
        {{- end }}
      labels:
        app: kafka-sample
    spec:
      serviceAccountName: {{ .Values.deployment.serviceAccountName }}
      containers:
      # hello world container
      - name: hello-world
        image: docker.io/b4bz/homer
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          {{ toYaml .Values.resources | nindent 10 }}
        env:
        - name: MY_ENV_VAR
          value: "my-env-var-value"
        - name: MY_ENV_VAR_2
          value: "my-env-var-value-2"
      restartPolicy: Always

default-namespace.yaml

Again, you can see the same values referenced in the namespace file, all defined in the values.yaml file.

YAML

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Values.namespace.name }}
  labels:
    app: {{ .Values.namespace.name }}

default-service.yaml

Thinking about conditional elements, imagine a situation like the one below where we are defining an exposing service. What if your local environment didn't support LoadBalancer type services? Docker Desktop, for example, doesn't support it. You could create an overrides file and adjust your template accordingly to allow you to select a different service type.

YAML

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.deployment.serviceName }}
  namespace: {{ $.Values.namespace.name }}
  labels:
    app: {{ .Values.deployment.serviceName }}
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: kafka-sample
  type: LoadBalancer

default-serviceaccount.yaml

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Values.deployment.serviceAccountName }}
  namespace: {{ $.Values.namespace.name }}
  labels:
    app: {{ .Values.deployment.serviceAccountName }}

Values.yaml

The values.yaml file contains the values for each of your defined variables.

YAML

deployment:
  name: homer-sample
  serviceName: homer-sample-service
  serviceAccountName: sa-homer-sample

namespace:
  name: homer-sample

conditionalAnnotations:
  enabled: true

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

We mentioned overrides earlier. These are supplementary files to the values.yaml file. They can be defined elsewhere and injected into the Helm command like this:

Bash

helm install kafka-sample-release ./homer-sample -f ./homer-sample-overrides/bld-override.yaml

In the example above, we are injecting an overrides file into the Helm command. This will take the values defined in that file and apply them on top of the values in values.yaml. An example of an overrides file is shown below.

Here, we are altering resource constraints for a lower-provisioned development environment.

YAML

resources:
  limits:
    cpu: 50m
    memory: 64Mi
  requests:
    cpu: 50m
    memory: 64Mi

And That's It

This covers everything I wanted to discuss in this post. I've found Kubernetes and Helm to be incredibly useful tools in my work. For smaller-scale applications, it might seem like overkill initially, but the effort invested in deploying an app this way can really pay off if it ever needs to scale.

How are you using Kubernetes and Helm in your hobby projects or even in your professional life? I'd love to hear more real-world examples of how people are using this fantastic technology.

This article covers the basics of setting up a private Jenkins instance (customised with Python) using Docker, setting up a Github PAT, registering that credential within Jenkins and configuring Jenkins to read a private repository.

Setting up Jenkins on Docker

I will use Docker Compose to set up the Jenkins environment.

I did these steps directly from the host that was running my 'prod' Docker instance, in my case its a Raspberry Pi 5 running Ubuntu.

I want to customise the Jenkins base image to include bash and python as my project is Django based. In order to achieve this I first create a folder on my Raspberry Pi called Jenkins and change directory into that folder.

mkdir jenkins
cd jenkins

The Dockerfile

Now I want to create a Dockerfile, I achieve this by running

vi Dockerfile

The contents of the that I created are below, as you can see I am creating this from the jenkins/jenkins:lts base image and then I am installing Python3 and associated tools.

FROM jenkins/jenkins:lts-jdk17

USER root

# Install Python 3 and related tools
RUN apt-get update && \
    apt-get install -y python3 python3-pip python3-venv bash && \
    apt-get clean

# Switch back to the Jenkins user
USER jenkins

Now create a file called docker-compose.yaml in the same folder as the above Dockerfile. and populate it like the below:

version: '3.8'
services:
  jenkins:
    build: . 
    user: "1000:1000"                  # UID:GID that exists on the host
    group_add: ["docker"]              # lets Jenkins run `docker build`
    restart: always
    ports:
      - 8090:8080
      - 50000:50000
    container_name: jenkins
    volumes:
      - /home/${myname}/jenkins_compose/jenkins_configuration:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock

${myname}: Create a .env file next to docker-compose.yaml with myname=<your-linux-user> or export myname before you run docker-compose. (Or simply replace this value in the docker-compose file before you run it with your Linux username)

Security note: The Compose file above used to set privileged: true in an older revision of this article. This hands the Jenkins container full root-level control of the host. That’s convenient for personal experiments, but it is not recommended for production or shared hardware.

In serious environments:

• Run Jenkins as an unprivileged UID/GID that matches a real user on the host.

• Give that user access to Docker only by adding it to the docker group (or by using a remote build agent), rather than running the container in --privileged mode.

• Mount /var/run/docker.sock read-only if the jobs merely build images; use root-less buildkit or a dedicated DinD sidecar if you need full write access.

Worth noting in this instance, I am using the build field rather than specifying a pre-created image. This means that when I run Docker Compose it will run the dockerfile that is in the same directory to build the image before deploying the image to a running container.

Starting Jenkins

Once this file is created you can run the command below to run it. I have added the --build argument to ensure the image is built each time rather than using a cached copy and also the -d arg to detach from the process

docker-compose up -d --build

You can then navigate to http://%your-docker-host%:8090 to start the setup on the Jenkins instance. For the purpose of this activity all the defaults are fine during the setup wizard. Don't forget to set a memorable username and password!

Once that is setup we are good to proceed onto the next step...

Configuring a PAT in Github and Jenkins

Next we need to create a Personal Access Token that will allow us to view our private repositories. I have opted to use a classic token for this purpose as I want my private Jenkins instance to be to see all my private repositories as I likely will expand the usage of this Jenkins instance.

In order to do this you follow the following steps on Github.com

• Log into your Github account

• Click on your profile in the top right hand corner then click settings

• Along the left hand menu, click 'developer settings'

• Click "personal access token" drop down

• Select "Tokens (Classic)"

• Click "Generate New Token" and choose the classic option

• Give it a name and also ensure "repo" is checked

  • Set the expiry time to a value desired

• Press Generate Token

• On the next screen ensure you copy the token and paste it somewhere safe for now. A text editor window is great for this

Once you have the PAT it is time to configure the Credentials with your shiny new Jenkins instance

Now for Jenkins config

To add this PAT to Jenkins navigate to your instance and login in

From Dashboard, click Manage Jenkins and then select 'credentials' from within in there click "System" > "Global Credentials" > "+ Add Credentials" the default option should be "Username and Password" and this is what we want.

• Keep Scope as default value

• username is your Github username

• Password is your PAT that you copied earlier

• ID can be anything to identify the credential

• Description can be filled appropriately

Now this is all setup for your first multibranch pipeline.

Setting up a multibranch pipeline for a private repo

From Jenkins Dashboard select "+ New Item" when the page loads give it a name and select "Multibranch Pipeline" and then click "ok".

• Give the pipeline a descriptive name

• Optionally give it a fitting description

• Select "Add Source" and Choose "Github"

• In credentials, in the drop down select the credential you just created

• Get your Repositories HTTPS clone link and paste it in

• As strategy I selected "Exclude branches that are filed as PRs"

• Under Build Configuration I set mode to 'by jenkinsfile' and set script path to Jenkinsfile - this is configurable depending on your needs as you can embed the script directly in the pipeline. I would opt to include it in your repo as you can change behaviour on an individual branch level

• I left the rest of the options as default

Once this is done, we should have a working pipeline, in order to test navigate back to "Dashboard" and select the pipeline that should now be visible. Once the pipelines homepage loads you should be able to select "Scan Repository Now" and this should give you some positive responses and may even populate branches and PRs depending on the state of your repo.

This will not work if you don't currently have a jenkinsfile in the repo if you followed the above configuration.

A simple Jenkinsfile example for a python Django project:

pipeline {
    agent any

    environment {
        // Define environment variables here, like your Python version
        VENV_DIR = 'venv'
        DJANGO_SETTINGS_MODULE = 'my-django-project.settings' // Adjust this to your settings module
    }

    stages {
        stage('Setup') {
            steps {
                script {
                    // Create a Python virtual environment
                    if (!fileExists("${VENV_DIR}/bin/activate")) {
                        sh 'python3 -m venv ${VENV_DIR}'
                    }
                }
                // Activate the virtual environment and install dependencies
                sh '''
                    . ${VENV_DIR}/bin/activate
                    pip install -r src/requirements.txt
                '''
            }
        }

        stage('Run Tests') {
            steps {
                // Run Django tests
                sh '''
                    export $(grep -v '^#' .env | xargs)
                    . ${VENV_DIR}/bin/activate
                    cd src
                    pytest --junitxml=../test-results.xml
                '''
            }
        }
    }

    post {
        always {
            // Archive test results and logs
            archiveArtifacts artifacts: 'test-results.xml', allowEmptyArchive: true
            junit 'test-results.xml'
            cleanWs()
        }
        failure {
            // Notify if the build fails
            echo 'The build failed.'
        }
        success {
            // Notify if the build succeeds
            echo 'The build was successful.'
        }
    }
}

This concludes this post, I hope it was of some use to someone!

The problems I have encountered arise when I want to test these applications locally, certain dependencies when included change how for instance I connect to a database through the introduction of properties such as

spring.cloud.gcp.sql.database-name
spring.cloud.gcp.sql.instance-connection-name
spring.cloud.gcp.sql.enable-iam-auth

Merely including the dependency of:

<dependency>
   <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>spring-cloud-gcp-starter-sql-postgresql</artifactId>
      <version>5.0.4</version>
   </dependency>
</dependency>

Makes the aforementioned properties required. Which is great and not a problem when my target is GCP. What if I want to test my code locally with a local instance of PostgreSQL?

Using profiles

We can make use of maven profiles to selectively include that dependency and also to ensure we choose an appropriate application.properties file for local running.

Initially I was looking at excluding a certain dependency for a certain profiles, but the answer appears to be to do the opposite.

It's fairly straight forward to set up in your pom.xml file

...
<!-- remove the dependencies from your higher level <dependencies> -->
<profiles>
    <profile>
        <id>gcp</id>
        <properties>
            <activatedProperties>gcp</activatedProperties>
        </properties>
        <dependencies>
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>spring-cloud-gcp-starter-sql-postgresql</artifactId>
                <version>5.0.4</version>
            </dependency
        </dependencies>
    </profile>
    <profile>
        <id>local</id>
        <properties>
            <activatedProperties>local</activatedProperties>
        </properties>
    </profile>
</profiles>
...

In my src/main/resource folder I have three files defined:

• application.properties
• application-gcp.properties
• application-local.properties

application.properties just contains a single line

spring.profiles.active=@activatedProperties@

For me the main difference is the properties that are used to connect to the DB

For GCP I am needing to provide the following properties in my application-gcp.properties file:

spring.cloud.gcp.sql.databanse-name=${DB_NAME}
spring.cloud.gcp.sql.instance-connection-name=${GCP_CONNECTION_NAME}
spring.cloud.gcp.sql.enable-iam-auth=true

For my application-local.properties file I am using more direct properties:

spring.datasource.url=${DB_HOST}

To select each profile, you can run a command such as

mvn package -P local

This was a useful way of leveraging maven profiles for me to be able to test my code locally and in GCP.

Conclusion

This is a helpful mechanism to use when you are wanting to be able to test your applications locally but they are destined to be deployed in the cloud. Being able to test changes locally is important and can help avoid wasting time later by repeatedly needing to deploy your application code to the remote environment every time an issue is encountered.

Cool, and?

Containers are a way to create a distinct operating environment for your applications by creating a sandbox that is lightweight and efficient. It shares the same kernel as the host system however, it runs in its own contained environment.

It was originally based on LXC containers for its runtime environment. LXC allows creating many different user space sessions using cgroups (control groups). These environments cannot interact directly with each other (they can use networking communication). Docker has moved on from this as a base technology however, the concepts remain the same.

This is how containers differ from virtualisation, they don't attempt to virtualise at the hardware level, instead, they do it at the user space level. This means a container is far lighter resource wise than a virtual machine. One knock on affect of this efficiency is containers are very quick to start up, very useful if you need to scale an application to answer a surge in demand!

Why should I care about it?

Its a very useful technology, it allows us to create images that can be used to create containers. The behavior of these containers can be a lot more predictable as the environment is defined when the image is created. You can take the same image and run it on any host and expect the same results. (Images are created through the use of a Dockerfile).

Any application can be containerized, and Windows containers are also possible although, typically, Linux is the target platform when containerizing an app.

Wait, windows can be containerized?

While I've not used it directly myself, I'm aware that Docker Engine - Enterprise allows the creation of Windows containers via namespace isolation technology allowing containers to share the same Windows Kernel.

From what I gather, Windows containers are less flexible than Linux containers; in the sense that they must be running the same OS version as the host. A Linux container can run any number of distributions of Linux. As a side note, you can run Linux containers via WSL.

How does this benefit in the real world

Containerised workloads can be leveraged to offer many advantages, some examples include:

Continuous Integration and Continuous Deployment (CI/CD)

We can make use of CI/CD tools such as Jenkins or Github Actions to create automated deployment pipelines. Typically this might look something like this. A change is made to an application and committed to the repository, commit triggers a Jenkins pipeline that performs unit tests / vulnerability scanning and container image creation (as a basic example). The container is then deployed to a container registry where it can be used to deploy to a production environment.

Microservice Architecture

Containers facilitate systems written with microservice architecture. Microservices ensure that functionality that makes up a larger system is decoupled, unlike monolithic applications. Some advantages of adopting a microservice pattern are:

• individual parts of the application can be updated without impacting other parts, reducing risk of introducing regression issues

• Individual parts of the application can be scaled independently, if a sales system gets a sudden surge of orders, the part that handles processing those orders could be scaled whereas other parts of the system could be left. This has large cost benefits for companies as its more surgical and less brute force!

Legacy applications modernisation

Containers provide a convenient way to create environments that allow legacy applications to run on more modern infrastructures. Even if it is monolithic. Whilst I would advocate for looking into refactoring or recreating if this isn't possible, containerisation alone does offer benefits. Containerising an application in this way can be the first step in a large modernisation project.

A brief look at a docker file

A Dockerfile can be as simple as:

FROM python:3.10.13-slim-bullseye

# Use apt-get to install system packages (we are using a Debian base image)
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    libc-dev \
    curl \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Copy the application code and requirements file from the code repository to a location inside the image
COPY ./app /app
COPY ./requirements.txt /requirements.txt

# Acquire the rust setup script and run it as well as set up the env var required
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

# Install Python dependencies using pip
RUN pip install --upgrade pip && pip install -r /requirements.txt

# WORKDIR is analogous to cd command
WORKDIR /app

# Run the app
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

This particular file is containerising a FastAPI (Python) application. It demonstrates how you can run commands to install system packages as well as using curl to install Rust runtime (required for one of the dependencies) and adding a path to the PATH env var. The CMD directive at the bottom of the file runs the uvicorn server and serves the application on part 8000. To create an image from this you would simply run:

docker build . -t name-of-image

And that's it, your application is containerized!

To Conclude

This article was intended to be a brief introduction to containers to anyone who might have heard about the tech before but didn't really have chance to work directly with it or fully appreciate its implications on how we might build systems.

If it has stoked your interest definitely start checking Docker out, it is super-easy to install via Docker Desktop and you can get started with containerizing your applications simply by adding a Dockerfile to your project and adding a few lines to that.

Do you have use cases that can't use containers? What about some unique situations where containers have been used? I'd love to hear about them!

Alembic is a database version control framework closely associated with SQLAlchemy, an independent Python ORM. I have used ORMs with Python in the past but more specifically it has been the one that is built into Django. For recent projects I have made a point of exploring different frameworks, specifically those with less included out of the box.

This lead me to FastAPI, a RESTful API framework for python. A couple of stand out features that has encouraged me to explore it more are its speed and the build in support for OpenAPI Specifications. This lead to the very valid question of how to handle my database layer and that introduced me to SQLAlchemy, that doesn't include version control as part of its core offering. I am a big fan of DB Version Control for very obvious reasons and the desire to have the functionality lead me to Alembic.

Project setup

The set up is pretty straight forwards, the first thing to ensure is that we have the required dependencies installed. I like to keep these to Python Virtual Environments because it makes sense to scope dependencies to a project.

$ mkdir db-tutorial
$ cd db-tutorial
$ python3 -m venv my-venv
$ source my-venv/bin/activate
$ pip install sqlalchemy alembic psycopg2-binary

We are creating a new project directory above and then creating a virtual environment using the "venv" module, we are calling the within it then activating it. Once that is done, we are getting the required dependencies from pip.

Hint: I tend to freeze the dependency list into a requirement.txt file when I install something new, this is important for collecting the correct dependencies down the line

$ pip freeze > requirements.txt

We are now good to go.

Initialising SQLAlchemy and Creating a Model

Initialise

First task is to do the require configuration for SQLAlchemy.

Lets create a module in at the root of our application that will hold our DB Connection Config and a model

$ mkdir persistence
$ cd persistence
$ touch __init__.py
$ touch db.py
$ touch models.py

open db.py in your favourite editor:

from sqlalchemy import Column, Integer, String, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import os

# Saving connection details in environment variables
# note storing password in plain text in this way would be bad in a production
# system
DB_USER = os.environ.get("DB_USER")
DB_PASSWORD = os.environ.get("DB_PASSWORD")
DB_HOST = os.environ.get("DB_HOST")
DB_PORT = os.environ.get("DB_PORT")
DB_NAME = os.environ.get("DB_NAME")

# Build the connection string
SQLALCHEMY_DATABASE_URL = f"postgresql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}"

# create the database connection
engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={})
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

# Get the base model class - we can use this to extend any models
Base = declarative_base()

# Provide a mechanism to retrieve the database from within the rest of the application
def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

The above code is essential boilerplate to configure SQLAlchemy. I have not hard coded any connection details in this snippet and instead would define them as Environment Variables in my run profile of VSCode (Or in a container).

Create a basic model

from sqlalchemy import Column, Integer, String
from persistence.db import Base
from sqlalchemy import  Column, Integer

# a model to hold car data
class Car(Base):

    __tablename__ = "cars"
    id = Column(Integer, primary_key=True, index=True)
    make = Column(String)
    model = Column(String, unique=True, index=True)
    colour = Column(String)

The above is a basic model that will serve the purpose of this tutorial

Note - At this point it is worth noting that I will use a Postgres database running within WSL but you aren't restricted to that for your backend - if SQLAlchemy supports it you can use it.

We are now in a good place, we have created the code to initialise the database connection and created a model to apply to it.

Init alembic and config

Now we want to init our version control system for our Database, lets start by opening a terminal and ensuring we are in the root of the project

$ alembic init alembic

This will initialise the alembic config, it creates a folder called "alembic" in the project root. It will also create "alembic.ini" in the same place.

alembic.ini contains a property that can be manipulated to provide the connection string of your DB to the framework however, I don't think this is the best way to go, we have already defined our connection string as part of our SQLAlchemy config and we are wanting to use the values stored in Environment Variables when the application is started. As such, we will defined our connection string in the generated /alembic/env.py file

from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
# import your DeclarativeBase
from persistence.db import Base  # adjust this to fit your actual module path
from persistence.db import SQLALCHEMY_DATABASE_URL

from persistence.models import Car

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# set to connection string defined in the db_base.py file
config.set_main_option("sqlalchemy.url", SQLALCHEMY_DATABASE_URL)

target_metadata = Base.metadata  # use your DeclarativeBase's metadata

Mostly this file can stay how it is when generated however there are a couple of elements that need considering:

• As discussed above, we are setting the sqlalchemy.url property programmatically to ensure we don't repeat thing to that end, we are importing SQLALCHEMY_DATABASE_URL from from db.py

• We are also importing our models to ensure Alembic can see them

Once this is done we can try and make the initial commit to the versioning

$ alembic revision --autogenerate -m "initial"

All being well, outputs with messages similar to

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'cars'
INFO  [alembic.autogenerate.compare] Detected added index ''ix_cars_id'' on '('id',)'
INFO  [alembic.autogenerate.compare] Detected added index ''ix_cars_model'' on '('model',)'
  Generating /home/james/python/db-tutorial/alembic/versions/0d25ea6c8183_initial.py ...  done

This has not yet created the table in the database but it has committed it to an internal table that alembic uses to track versioning and has created the migrations under the alembic/versions folder. If you look into that file created in there you will see your models being described.

To push the changes to the database you can use the command

$ alembic upgrade head

If you look inside your database, you will see that a cars table has been created.

The takeaway

Your SQLAlchemy based database schema is now version controlled. Not only can you apply migrations as you make changes to your models but you can also go backwards too

$ alembic downgrade -1

would take you back one revision, if you put -2 it would go back two revisions and so on and so forth.

In conclusion, we've successfully navigated the landscape of database version control using Alembic, a robust framework closely entwined with the powerful SQLAlchemy ORM.

By venturing beyond the familiar confines of built-in Django ORM, we've embraced the flexibility of FastAPI and harnessed its speed and native support for OpenAPI Specifications. With a meticulous project setup, we crafted SQLAlchemy configurations, initialized database connections, and defined models for our journey.

The initiation of Alembic brought version control to our database schema, offering not only seamless migrations but also the ability to gracefully navigate backward when needed. As we delve deeper into the realm of SQLAlchemy and Alembic with FastAPI, this tutorial serves as a foundational guide, laying the groundwork for reliable, iterative application development. Stay tuned for more insights and explorations in the exciting intersection of Python, databases, and web frameworks.

Thanks for joining in, and I hope you found this journey into database version control both enlightening and useful!

My favourite animal is the Cheetah and I also like birds of prey. I like a neon-themed desktop environment so I thought I'd test the Dall.E 3 capabilities and get it to generate some wallpapers for me. I'm quite satisfied with the results. Sharing them in case anyone else would like to use them. I do admit, it's a random combination!