Go, Python, PHP, JavaScript, Ruby

Google Cloud Platform (GCP), Amazon Web Services (AWS), Docker, Kubernetes, Terraform, Pulumi, GitLab CI/CD, GitHub Actions, GCP Cloud Run, GCP Cloud Storage, AWS AppRunner, AWS EC2, AWS Batch, AWS Lambda, AWS S3, RabbitMQ, Auth0, Datadog, ELK, Sentry, New Relic, Debian, Linux, MongoDB Atlas

PostgreSQL, MongoDB, AWS Neptune, Redis, SQlite, MySQL/MariaDB, Elasticsearch

GCP Vertex AI, GCP NotebookLM Enterprise, OpenRouter, Opik, Langfuse, Open WebUI, AWS SageMaker, LLM

REST, Microservices, SQL, NoSQL, CI/CD, git, middleware, ODRL, turtle, RDF, SPARQL, WebSockets, Clean Architecture, OpenAPI, JSON-LD, JWT, TDD, SCRUM, DDD, MVC, Rails, CQRS, OOP, ISTQB, Risk-Based Testing, BDD, API, CLI, CRUD, CSS, CSV, HTML, JSON, PDF, RFC, Dagster, Selenium, Tricentis Tosca

At the beginning, the AI strategies for detecting issues were largely untested. I conducted a series of experiments to gain a first understanding of their effectiveness. This included generating synthetic data, prompt engineering, setting up ETL pipelines in Dagster and running evaluations with Opik.

All scripting was done in Python. AI services were provided by GCP Vertex AI and OpenRouter.

Project documentation of this nature is highly confidential. To securely process data, I set up the initial services on VMs in a private VPC on Google Cloud Platform (GCP).

To access this private VPC I connected it to the company's Tailscale network with a subnet router that also acted as a nameserver for the subnet and featured an authentication proxy that provided access from Tailscale clients to the Google Cloud Platform (GCP) services, like GCP Cloud Run and GCP Cloud Storage. This setup was fully configured with Terraform.

Source code was managed in a monorepo with a fully automated GitLab CI/CD pipeline to push Docker images to the Google Cloud Platform (GCP) artefact registry, with deployment to various customer and developer environments.

Together with the first arrived permanent engineers, I started building the backend of the actual application in Python. My work focused on reporting on metrics like Precision, Recall, F1-score, etc. I also implemented similarity search and clustering in order to identify the set of unique findings for a particular dataset.

In the final setup, Opik was switched for Langfuse for observability. Dagster was dropped. The requirements for annotation and experimentation turned out to be special enough to warrant a custom solution.

ODRL is built on RDF because a knowledge graph is well-suited to express the rich data structures found in contracts. A populated graph can answer questions like "Under what conditions is this action allowed, and are they met?" But there is also a great need to manage all the different entities that are involved, like actions, permissions, duties, parties, prices, etc.

The challenge in managing all these entities was that the chosen graph store, AWS Neptune, does not offer any data integrity functionality. There is no schema, there are no transactions, etc. There is certainly no ORM for it. It really is just a big bucket of triples.

To help rapid development of various web applications, I developed a Go backend that directly interfaces with the SPARQL endpoints of AWS Neptune and that can be used as a simple REST/JSON CRUD store for the ~120 different types of ODRL entities. It rigorously validates the entities and makes sure the data always adheres to the model. It handles concurrent requests, uses JWT tokens for authentication, etc. It behaves like any other REST/JSON API, but if desired, it can also respond in JSON-LD.

Speedy iteration was achieved by constructing a fully automated CI/CD pipeline that used multiple interacting Docker containers, running in GitHub Actions, with thousands of unit tests and broad integration tests. The backend was automatically deployed to an AWS AppRunner instance in either a dev or prod environment. Identity management was handled by Auth0.

The main source of contract and pricing information were PDFs. Not thousands of them, but enough to make it very impractical to process them locally and share the results by sending around CSV files. In addition, it was necessary to provide references for the data that was extracted from these PDFs by locally running Python AI pipelines to the original documents, so the results could be verified.

Huge gains in productivity were possible by moving these functions to a central cloud platform and automating them as much as possible. However, this was an already running, essential, but also still changing process. Taking time to build a complete new platform was impractical. Instead, I proposed a bottom-up approach, where everyone could continue to work, but where step by step functions would be moved and/or automated.

Steps taken in the first phase:

• Define a file hierarchy that expresses types of documents, their relations, and their different stages of processing
• Move all documents to an AWS S3 bucket following that hierarchy
• Convert the Python AI pipelines into AWS Batch processes
• Develop a Go service on AWS AppRunner that
  • performs sanity checks on the files
  • triggers the AI pipelines
  • imports the resulting references into a PostgreSQL database
  • serves the references and the documents through an API

A colleague developed the frontend that would load the relevant documents and highlight the source of a data point. Full CI/CD was again implemented with Docker and GitHub Actions. Identity management was again handled by Auth0.

There was a great interest in having a LLM-powered chatbot that could be queried for contract data, even though LLMs can answer with incorrect information.

A series of prototypes was developed to find out the setup that would provide the most accurate answers. Current LLMs can handle a large enough context window to hold complete contract information in turtle format. Different setups were tried out:

• Open WebUI on GCP Cloud Run with GCP Cloud Storage and GCP Vertex AI
• GCP NotebookLM Enterprise

The prototypes focused on limited data sets. Scaling them up was postponed in anticipation of the coming Google AgentSpace (GCP).

The data scientists developed more advanced forms of aggregation with entity recognition. For instance, from a tweet reporting on a robbery, a collection of AWS SageMaker services could infer the time, location, perpetrator, method, and victim.

To present this information in the frontend, the challenges to overcome were:

• The AWS SageMaker services required careful batching for efficient processing
• The datamodel used in communication with the frontend was incompatible with this change

For the backend, we used a Pipes and Filters architecture that initially used Go channels to move data from one stage to the next. Later a few of them were migrated to RabbitMQ queues for better resilience and to simplify monitoring.

The frontend incompatibility was solved by facilitating regular communication with the frontend team and iterating together, achieving a seamless transition to a new data model.

When the company started to use MongoDB as a storage backend, there was no official Go driver, and there weren't plans to release one. As a result, the community created several drivers, among which GlobalSigns mgo, which was used in about 50 services on the platform.

Later, Mongo did release an official driver, and when the company wanted to switch to MongoDB Atlas for managed hosting, all services had to be converted to that driver, as this was a requirement of the new service.

The two drivers were incompatible, and all ~50 services used it all in slightly different ways, making it impossible to automate the conversion. The process was helped by setting up a wiki with extensive documentation on what to do in each situation and a growing list of examples, but in the end, it was simply grunt work done with a small group of developers.

These days, it would probably be an excellent job for AI.

The company was growing, but not all processes had matured at the same speed. Major technical decisions were still done the same way as when the company started: a Slack video call where everyone who might be interested, however remotely, was invited to join. There was little to no preparation for these meetings. Someone simply stated a question or a problem at the start, and brainstorming would ensue.

As can be imagined, this resulted in very haphazard decision-making.

Feeling unsatisfied, I researched how other projects that worked remotely and asynchronously did their architectural design and was inspired by the RFC process used by the creators of the internet itself.

After getting permission to pursue the idea was given by management, I designed and implemented a light-weight version of it. This was met with great enthusiasm and success. The introduction of the system kept showing up in surveys on employee satisfaction for the coming years as an example of a positive change in the company.

There were a variety of monitoring tools, used both on public and private clouds, that needed to be connected to notification services in an SLA-aware manner. If an incident happened in the middle of the night, but the customer had an SLA that specified service windows would start at 9:00 AM on workdays, the notification had to be delayed until the next day, as to not needlessly page the on-call SRE while sleeping.

The middleware was developed in Go and used MongoDB for storage and synchronisation between data centers. The solution used RabbitMQ for queueing and could be called from the monitoring tool via a webhook, or a local CLI app could be used to forward the events.

This was done by a team of two and was my first taste of production-level Go, with a distributed service, extensive automated tests, and full CI/CD with GitLab CI/CD.

The system was responsible for ticketing, invoicing, a CMDB, time tracking, some CRM, and much more.