Crumb
c/selfhosted · by chrash0@lemmy.world · 250d

what do y'all use for CI/CD?

hey nerds! i got a lovely email from GitHub this morning that their increasingly vibe-coded, barely-working Actions features are about to get more expensive (charging by the minute for something that notoriously spin-locks is a special flavor of shit sandwich).

i usually just use whatever i’m given at wherever i’m working. i do have a project that i maintain to parse Ollama Modelfiles tho: https://github.com/covercash2/modelfile and to be honest, Actions is the only solution i’ve ever used that came close to sparking joy, simply because it was easy to use and had tons of community mind-share (i’ve definitely heard horror stories and would never stake my business on it), but this price increase and all the other news around GitHub lately has got me side-eying self-hosting solutions for my git projects. Forgejo seems like the way to go for git hosting, but Actions in particular Just Works™️ for me, so i’m kind of dreading setting something up that will be yet another time sink/rabbit hole (just in time for the holidays! 🙃).

i can install most of my tooling with my language toolchain (read: rustup and cargo) which makes things fairly neat, but i just don’t have a sense for what people use outside of Jenkins and Actions.

i thought this community might have some insight beyond the LLM generated listicles that have blighted modern search results.

thanks in advance 🙏

103 points · 76 comments · view on lemmy.world

76 Comments

h54@programming.dev · 36 pts · 250d (5 replies)

IMO, Gitlab CI/CD blows Github out of the water. They're not even in the same league. I recommend Gitlab + self hosted runners (it's so easy).

I've been using Gitlab for many years and host my own runners as of the past 6 months because I nearly exhausted my monthly free tier runner minutes one month.

xcjs@programming.dev · 12 pts · 250d

I second GitLab CI/CD - it's a CI/CD system that just makes sense to me. That doesn't mean it doesn't have its complexities depending on your needs, but I've overall enjoyed my time working with it.

douglasg14b@lemmy.world · 1 pts · 249d (1 reply)

Edit: I forgot this was self-hosted community, disregard.

How does organization work out?

We have dozens of workflows for our monorepo CI/CD stuff. GitHub organization with the flat structure is incredibly annoying.

GitLab is a single file?? (Or am I misinformed? )How does that work out?

h54@programming.dev · 3 pts · 249d

The repo specific config is a single file. You can also import templates/other files if need be. I worked in a shop where Devops set up a bunch of templates for generic, common jobs which made getting started easy. If custom config/code is required, overriding a templated job was easy. I was responsible for migrating my team's ~50 repos (services, libraries, etc) from Jenkins + Bitbucket into Gitlab and found it to be pretty straightforward.

jacksilver@lemmy.world · 1 pts · 249d (1 reply)

I had someone swear to me that Github templating was better, but I've only worked with Gitlabs templates. Why do you like Gitlab over Github?

h54@programming.dev · 5 pts · 249d

Gitlab CI feels native. Github offers similar functionality but it feels/looks like an afterthought. I think the Gitlab .yaml structure is more intuitive. Also, how the Gitlab UI visually represents a pipeline is mcuh better, IMO. Self hosting runners on my server (Ubuntu) is so easy and free. I hadn't tried it with Github but it sounds like it still costs money?!

Note: I don't work for Gitlab

Jayjader@jlai.lu · 30 pts · 250d (10 replies)

Forgejo has their own runner: https://forgejo.org/docs/latest/admin/actions/runner-installation/

I've used it on my personal machine, was very easy to setup and mostly compatible with GitHub actions out-of-the-box (including things like actions/checkout@v4).

scrubbles@poptalk.scrubbles.tech · 16 pts · 250d (4 replies)

Forgejo runners are great! I found some simple actions to do docker in docker and now build all my images with them!

felbane@lemmy.world · 1 pts · 249d (3 replies)

please share, I'm interested in doing the same

scrubbles@poptalk.scrubbles.tech · 2 pts · 249d (2 replies)

Sure! I use Kaniko (Although I see now that it's not maintained anymore). I'll probably pull the image in locally to protect it...

Kaniko does the Docker in Docker, and I found an action that I use, but it looks like that was taken down... Luckily I archived it! Make an action in Forgejo (I have an infrastructure group that I add public repos to for actions. So this one is called action-koniko-build and all it has is this action.yml file in it:

name: Kaniko
description: Build a container image using Kaniko
inputs:
  Dockerfile:
    description: The Dockerfile to pass to Kaniko
    required: true
  image:
    description: Name and tag under which to upload the image
    required: true
  registry:
    description: Domain of the registry. Should be the same as the first path component of the tag.
    required: true
  username:
    description: Username for the container registry
    required: true
  password:
    description: Password for the container registry
    required: true
  context:
    description: Workspace for the build
    required: true
runs:
  using: docker
  image: docker://gcr.io/kaniko-project/executor:debug
  entrypoint: /bin/sh
  args:
    - -c
    - |
      mkdir -p /kaniko/.docker
      echo '{"auths":{"${{ inputs.registry }}":{"auth":"'$(printf "%s:%s" "${{ inputs.username }}" "${{ inputs.password }}" | base64 | tr -d '\n')'"}}}' > /kaniko/.docker/config.json
      echo Config file follows!
      cat /kaniko/.docker/config.json
      /kaniko/executor --insecure --dockerfile ${{ inputs.Dockerfile }} --destination ${{ inputs.image }} --context dir://${{ inputs.context }}

Then, you can use it directly like:

name: Build and Deploy Docker Image

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: docker

    steps:
    # Checkout the repository
    - name: Checkout code
      uses: actions/checkout@v3

    - name: Get current date # This is just how I label my containers, do whatever you prefer
      id: date
      run: echo "::set-output name=date::$(date '+%Y%m%d-%H%M')"

    - uses:  path.to.your.forgejo.instance:port/infrastructure/action-koniko-build@main # This is what I said above, it references your infrastructure action, on the main branch
      with:
        Dockerfile: cluster/charts/auth/operator/Dockerfile
        image: path.to.your.forgejo.instance:port/group/repo:${{ steps.date.outputs.date }}
        registry: path.to.your.forgejo.instance:port/v1
        username: ${{ env.GITHUB_ACTOR }}
        password: ${{ secrets.RUNNER_TOKEN }} # I haven't found a good secret option that works well, I should see if they have fixed the built-in token
        context: ${{ env.GITHUB_WORKSPACE }}

I run my runners in Kubernetes in the same cluster as my forgejo instance, so this all hooks up pretty easy. Lmk if you want to see that at all if it's relevant. The big thing is that you'll need to have them be Privileged, and there's some complicated stuff where you need to run both the runner and the "dind" container together.

felbane@lemmy.world · 1 pts · 249d (1 reply)

Thanks for the write-up! I've been trying and failing to do DOOD and POOP runners via forgejo, but I haven't had the time or energy to really dig in and figure out the issue. At this point I just want something to work so I'll give your setup a try 😎

scrubbles@poptalk.scrubbles.tech · 1 pts · 249d

Of course! Let me know how you run your containers and I may be able to help on that side too

corsicanguppy@lemmy.ca · -6 pts · 250d (4 replies)

It's still yaml shit though.

zr0@lemmy.dbzer0.com · 6 pts · 250d

Every language, that uses functional white spaces, is absolutely awesome!!

— no one

prettybunnys@piefed.social · 3 pts · 250d

What issue do you have with using yaml to define a job?

Jayjader@jlai.lu · 2 pts · 250d (1 reply)

I dislike yaml as much as the next person, but you can always "just" write Jason JSON (lol autocorrect). Unless I'm misunderstanding your criticism?

chilicheeselies@lemmy.world · 2 pts · 249d

Yaml is vette than json for this IMO brcausebyou can write comments in yaml, and in general format multiline strings easier. Json is best for system to system comms. Human to system literlaly anything other text formst than json.

thesmokingman@programming.dev · 14 pts · 249d

Please don’t take me as a GH shill because I’m not. I’m not sure we read the same email given your projects. Actions on GH runners are dropping in cost and there’s a new fractional cost for self-hosted. For the average user, especially those on GH runners, costs are going down. Looking at your repo, you haven’t run anything since July. Your workflow files use GH runners. Nothing in your history suggests you’re leaving the free tier so I don’t get this FUD at all. General Microsoft hate? Fuck yeah. Shitty GH service? Fuck yeah. Plenty of reasons to dunk but this was not one of them. M

Carol2852@discuss.tchncs.de · 14 pts · 250d (15 replies)

I'm using gitea which has CI compatible to GitHub actions with my own runner. It's pretty straightforward to set up and didn't give me any headaches yet. It's a very small instance just for my ownaybe dozen projects though.

yaroto98@lemmy.world · 6 pts · 250d (10 replies)

This is what I was using till I switched to forgejo and never got around to setting up one of their runners.

cecilkorik@lemmy.ca · 2 pts · 250d (1 reply)

If it helps motivate you to give it a shot, I found gitea's runner very confusing to set up, but I felt like forgejo was better designed, pretty easy and well documented.

chrash0@lemmy.world · 1 pts · 250d

heck yeah this is the review i was looking for 💯

Carol2852@discuss.tchncs.de · 2 pts · 249d

I run their act binary on one of my servers. Can't remember much of the setup, so I can't be too bad. I did have to change the used images though, but I guess that comes with maintenance of you own runner anyway.

witten@lemmy.world · 1 pts · 249d (2 replies)
[ removed ]
yaroto98@lemmy.world · 1 pts · 249d (1 reply)

I hadn't used gitea for long. I just had both running, and then cloned my repos one at a time manually. So long as I had the code, I didn't really care.

witten@lemmy.world · 1 pts · 249d
[ removed ]
finn@toot.fan · 1 pts · 249d (3 replies)

@yaroto98 @Carol2852 Same here, switched from gitea to forgejo. I still was using the act runner for some time, later I replaced it with the forgejo-runner.

Works pretty smooth!

witten@lemmy.world · 1 pts · 249d (2 replies)
[ removed ]
finn@toot.fan · 1 pts · 249d (1 reply)

@witten I have switched maybe a year ago or something like that. Didn't loose anything because I was running a compatible version at that time:
https://forgejo.org/docs/latest/admin/upgrade/from-gitea/

If you are running a recent version it's probably a bit more complicated.

witten@lemmy.world · 1 pts · 249d
[ removed ]
corsicanguppy@lemmy.ca · 3 pts · 250d (2 replies)

CI compatible to GitHub actions

Ugh. More yaml?

chocrates@piefed.world · 3 pts · 249d (1 reply)

I get the hate but did you ever have to maintain jenkins pipelines? I'll take yaml any day.

I'm game to explore the next evolution though.

Carol2852@discuss.tchncs.de · 1 pts · 249d

That was my first thought as well. 😁

chrash0@lemmy.world · 2 pts · 250d

good lead. it’s just the one project for now, and to my surprise it’s actually a dependency for the ollama-rs project, so i feel somewhat obligated to keep it stable.

Routhinator@startrek.website · 12 pts · 249d

Forgejo and self hosted action workers.

avidamoeba@lemmy.ca · 10 pts · 250d (7 replies)

Jenkins

prettybunnys@piefed.social · 9 pts · 250d (5 replies)

Jenkins is good enough to be widely used enough to be hated enough to be downvoted.

The sign of a mature product IMO.

You could do worse than Jenkins

avidamoeba@lemmy.ca · 5 pts · 250d (1 reply)

Been using Jenkins since before it was called Jenkins. It's been in use at every corpo I've worked for. It can practically do anything. Especially coupled with Docker.

elephantium@lemmy.world · 4 pts · 249d

Hudson? Man, that's a blast from the past.

PushButton@lemmy.world · 3 pts · 249d (1 reply)

I was scrolling, looking for a Jenkins somewhere, to finally find that post with down votes.

Every company I go, it's a different CICD, and they all make me wish to use Jenkins instead.

prettybunnys@piefed.social · 1 pts · 249d

Jenkins is better than many but IMO Gitlab pipelines are top tier.

synae@lemmy.dbzer0.com · 1 pts · 249d

"It's the worst one, except for all the others"

AA5B@lemmy.world · 3 pts · 249d

I’m not entirely sure why all the hate : Jenkins can do the most things the must ways. And yes, it’s so much nicer defining a pipeline with a fully functional language than an assortment of yaml files

Actually that was my response when my company wanted to start using Gitlab ci. It only has one way of doing things so you can probably get a faster start if you had no ci, were a small company, and had simple builds. However we’re over 4,000 builds in many languages from 12 year old monoliths to modern micro services and containers….. and way too much godawful JavaScript. Do you want the quick and simple tool great for a small startup or the all powerful kitchen sink of tools?

irmadlad@lemmy.world · 10 pts · 250d (3 replies)

Watching this thread because CI/CD is something that I'd like to get into.

Zagorath@aussie.zone · 3 pts · 250d

Ditto

elephantium@lemmy.world · 2 pts · 249d (1 reply)

Are you a programmer?

irmadlad@lemmy.world · 3 pts · 249d

I..uh....I pretend I am from time to time.

EarMaster@lemmy.world · 9 pts · 250d

Gitlab CI/CD pipelines are my go-to tool. At work we self host an instance, for personal projects I use gitlab.com.

cosmicrose@lemmy.blahaj.zone · 9 pts · 250d

I self-host https://woodpecker-ci.org/ and I love it. It was easy to set up, and I never have to worry about CI/CD minutes.

_stranger_@lemmy.world · 9 pts · 250d (1 reply)

Magnetic needle. Steady hand.

TechieDamien@lemmy.ml · 6 pts · 250d

Not butterflies?

trougnouf@lemmy.world · 8 pts · 249d (1 reply)

Self-hosted Forgejo Actions on a Codeberg repository. It was relatively easy to setup and I don't even need a VPS through my dynamic IP 5G connexion. See also: https://codeberg.org/trougnouf/cfait

victorz@lemmy.world · 4 pts · 249d

connexion

I'm imagining you saying "connex-yun", and it reminds me of Stewie saying "cool-hhhwip".

fruitycoder@sh.itjust.works · 6 pts · 249d

Git lab CI is my goto for git repo based things (unit tests, integration tests, etc). Fleet through Rancher for real deployments (manages and maintains state because kubernetes). Tekton is my in between catchall.

elephantium@lemmy.world · 5 pts · 249d (2 replies)

We use Azure Devops at my current gig. It works pretty well for our setup. I've used GHA before; it definitely didn't "spark joy". I wastedspent way too many hours in the "update yaml file, commit, push, wait 5 minutes for it to fail again" spiral of despairfeedback loop.

Nice thing with ADO is its release dashboard -- you get a really nice summary of recent builds and where they went:

$project - dev - test - prod

I didn't see anything similar for GHA.

douglasg14b@lemmy.world · 1 pts · 249d (1 reply)

A lot of that pain can be reduced by writing and running your code locally before pushing it to a CI environment. Generally with our automation we write a CLI, And GitHub actions is just an execution environment that calls the CLI.

And if what you're trying to do must execute inside an action. You can run workflows locally with docker!

elephantium@lemmy.world · 1 pts · 249d

That's a great idea if it's possible, but I want to say it wouldn't have helped with our environment at the time.

I almost wish I could look back at that repo and share the yaml file here, maybe I was missing something back then. I'm certainly more proficient with yaml now.

I do recall wishing there was a way to simulate the execution locally. I think I remember hearing about a local runner, but it had too many caveats to help.

possiblylinux127@lemmy.zip · 5 pts · 249d

Woodpecker CI

eager_eagle@lemmy.world · 4 pts · 250d (4 replies)

fwiw, you can self host a GitHub actions runner

Natanox@discuss.tchncs.de · 4 pts · 250d (2 replies)

Don't they want to monetize those as well?

chrash0@lemmy.world · 8 pts · 250d

yes, according to this morning’s email

eager_eagle@lemmy.world · 1 pts · 250d

ah right, my bad

scrubbles@poptalk.scrubbles.tech · 2 pts · 250d

But you are charged for it.

weastie@lemmy.world · 4 pts · 249d (1 reply)

I do devops at work and my experience is that really any CI/CD system works, they all have enough features to do what you want. They all fundamentally just run scripts on boxes. Therefore, I say pick the easiest one, likely the one that is built into whatever Git system you are using.

Try to keep your pipelines simple-ish when you can, they almost never need to be that complicated. 95% of the time it's just running a command or two. If a pipeline needs to do something complex, I'd recommend writing that script into the Git repo and calling it, rather than having a CI job that is 100 lines long.

chrash0@lemmy.world · 1 pts · 249d

this is my experience as well. we have a bespoke wrapper around Jenkins, and the more we can test locally the less time we have to spend waiting for the system to fail. it’s one of the reasons i’ve adopted just to script things locally as if it was CI.

michael@piefed.chrisco.me · 3 pts · 250d (1 reply)

Used to use travis or clicleci and they both worked really well. Theres some issues with travis being old/expensive and circle got in touble for a few security issues though. gitlab has some nice tools from my experience.

Im interested as well. Ive got a forgjo that I would love to hook into at some point.

iatenine@piefed.social · 1 pts · 250d
[ removed ]
killabeezio@lemmy.zip · 3 pts · 249d (2 replies)

So many these days. Actions are probably one of the best, but there are still plenty of others out there.

  • gitlab
  • dagger
  • concourseci
  • tekton
  • Spinnaker
  • harness
  • argo
  • flux
  • gocd

If I were to pick one, it would probably be dagger. Or really anything but Jenkins.

partofthevoice@lemmy.zip · 2 pts · 249d

Gitea Actions, as well.

chrash0@lemmy.world · 1 pts · 249d

we use Jenkins + a bespoke wrapper at work. thats left a bad taste in my mouth enough to avoid Jenkins altogether

verstra@programming.dev · 3 pts · 249d

I'm currently looking into Concourse.

It does have steeper-than-average learning curve, but I really like that it has well-defined fundamentals (resources, jobs, tasks) and isolation with OCI containers. Before I adopt it fully, I want it to run my nix flake dev shell.

iatenine@piefed.social · 2 pts · 250d (1 reply)

Are you sure it was a price "increase"?

I got a similar email this morning but it was the exact opposite of what I expected upon closer examination:

https://docs.github.com/en/billing/reference/actions-runner-pricing

chrash0@lemmy.world · 2 pts · 250d

you’re right. i just expected it to be an increase 😅

corsicanguppy@lemmy.ca · 2 pts · 250d

Where's the Bazel people at?

TheHolm@aussie.zone · 2 pts · 246d (1 reply)

Woodpecker. No BS CI which can be attached to pretty much anything. It just need a webhook and way to pull your project.

chrash0@lemmy.world · 1 pts · 246d

nice. simple and modular i like. i deal with far too many “one stop shops” at work to bring that home

eager_eagle@lemmy.world · 2 pts · 250d (1 reply)

btw, the prices of managed runners are going down, not increasing

https://docs.github.com/en/billing/reference/actions-runner-pricing#standard-github-hosted-runners

still good to have a self-hosted alternative though

chrash0@lemmy.world · 2 pts · 250d

i honestly didn’t look that close, obviously haha

but yeah, i’ve been kinda looking for a reason to de-Microsoft my stuff

douglasg14b@lemmy.world · 2 pts · 249d

GitHub Actions mostly.

The rest is usually plumbing and code to support it. The actions are just the automated execution environment.

Mihies@programming.dev · 1 pts · 250d

I use cake build to create the build script and then I can run it from wherever.

HelloRoot@lemy.lol · 1 pts · 250d

https://sourcehut.org/

slacktoid@lemmy.ml · 1 pts · 250d
[ removed ]