DSCI - CI engine with programming languages to write pipelines (no YAML)

Only as high level glue , main logic is just normal programming languages - see here https://github.com/melezhik/DSCI/tree/main/examples

I mean there is only so many ways of representing a data structurally clearly. I find that yaml is good enough but like XML leads to a lot of bloat, but unlike XML isn't nearly as complicated. I'm not sure making it a library resolves the issue better than creating a custom grammar that appropriately accommodates what people want to express in the language easily, make the possible and desired easy, the impossible and undesirable states hard if not unrepresentable. I mean having a custom language does allow for loops, and functions which isn't something I find I need too much in CI. Perhaps templating / functions but having a full Turing complete language leads the way for bad states / nonideal uses.

Here is SDK for those languages:

Raku Perl Bash Python Ruby Powershell Php Golang

UPDATE: so it's not just hitting up a language from a pipeline , it's full reach SDK, one writes a pipeline on a language, if I get your comment correctly

Ok, try to do it on GH actions, share states between tasks/jobs for example:

tasks/task_one/task.py

#!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

tasks/task_two/task.py

#!/usr/bin/python3

dict = get_state()
print(dict["out1"])
print(dict["out2"])

Or share states between jobs:

jobs/job1/task.py

#!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

jobs/job2/task.py

#!/usr/bin/python3

dict = config()

print(dict["_dsci_"]["job1"]["out1"])
print(dict["_dsci_"]["job1"]["out2"])

I can't imagine how much boilerplate code (if this ever possible ) one needs to write to achieve that on YAML based pipelines (GH Actions/ etc)

And don't tell me about jobs artifacts ))

UPDATE:

Another good example is to run tasks conditionally, yes using old good if:

#!/usr/bin/python3

if some_condition(foo, bar): 
    run_task(
       'task1', {
          'foo' : 'foo value',
          'bar' : 'bar value'
       }
    )

The same could be quite awkward in YAML based code